MATH 402 - LINEAR REGRESSION -
Least Squares Estimation
Question Bank - Set 10
Liberty University
Question 1
Question
Let Ydenote the response variable and Xdenote the predictor variable. Con-
sider the following sample data points:
X Y
1 2
2 3
3 5
4 7
5 8
Using the method of least squares, find the equation of the least squares
regression line to predict Ygiven X.
Solution
Step 1: Calculate the means of X(¯
X) and Y(¯
Y) as well as the deviations from
the mean for each data point (Xi−¯
X)and (Yi−¯
Y).
Given data points are:
X Y
1 2
2 3
3 5
4 7
5 8
Calculating the means:
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+8
5= 5
Deviations from the mean:
X(X−¯
X) (Y−¯
Y)
1−2−3
2−1−2
3 0 0
4 1 2
5 2 3
Step 2: Calculate the sum of squared deviations from the mean of X(SXX ),
the sum of squared deviations from the mean of Y(SY Y ), and the sum of
cross-products of deviations (SXY ).
SXX =
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ 02+ 12+ 22= 10
SY Y =
5
∑
i=1
(Yi−¯
Y)2= (−3)2+ (−2)2+ 02+ 22+ 32= 22
SXY =
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3)+(−1)(−2)+(0)(0)+(1)(2)+(2)(3) = 13
Step 3: Calculate the slope of the least squares regression line:
b=SXY
SXX
=13
10 = 1.3
Step 4: Calculate the intercept of the least squares regression line:
a=¯
Y−b¯
X= 5 −1.3(3) = 0.1
Therefore, the equation of the least squares regression line is:
ˆ
Y= 0.1+1.3X
Question 2
Question
Let X=
1 1
1 2
1 3
1 4
be the design matrix and y=
2
1
3
5
be the response vector.
Find the least squares estimate ˆ
βof the regression coefficients in the model
Y=Xβ +ϵ, where ϵis the error term.
2
Solution
Step 1: The least squares estimate ˆ
βis given by the formula:
ˆ
β= (XTX)−1XTy.
Step 2: First, calculate XTX:
XTX=(1111
1234)
1 1
1 2
1 3
1 4
=(4 10
10 30).
Step 3: Next, calculate (XTX)−1:
(XTX)−1=(4 10
10 30)−1
=1
4·30 −10 ·10 (30 −10
−10 4 )=1
20 (3−1
−1 0.8).
Step 4: Now, calculate XTy:
XTy=(1111
1234)
2
1
3
5
=(11
26).
Step 5: Finally, calculate ˆ
β:
ˆ
β=1
20 (3−1
−1 0.8)(11
26)=1
20 (8
2)=(0.4
0.1).
Therefore, the least squares estimate of the regression coefficients is ˆ
β=
(0.4
0.1).
Question 3
Question
Suppose we have a dataset with the following points: (1, 5), (2, 7), (3, 9), (4,
11). We want to find the least squares regression line for this dataset. Find the
equation of the regression line.
Solution
Step 1: First, we need to calculate the mean of x (¯x) and the mean of y (¯y).
¯x=1+2+3+4
4= 2.5
3
¯y=5+7+9+11
4= 8
Step 2: Next, we calculate the sum of the squares of the deviations of x and
y from their means.
4
∑
i=1
(xi−¯x)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5
4
∑
i=1
(yi−¯y)2= (5 −8)2+ (7 −8)2+ (9 −8)2+ (11 −8)2= 10
Step 3: Now, we calculate the sum of the products of the deviations of x
and y from their means.
4
∑
i=1
(xi−¯x)(yi−¯y) = (1−2.5)(5−8)+(2−2.5)(7−8)+(3−2.5)(9−8)+(4−2.5)(11−8) = 7
Step 4: Using the formula for the slope of the least squares regression line,
we have:
b=∑4
i=1(xi−¯x)(yi−¯y)
∑4
i=1(xi−¯x)2=7
5= 1.4
Step 5: Finally, we find the y-intercept of the regression line using the for-
mula:
a= ¯y−b¯x= 8 −1.4(2.5) = 4
Therefore, the equation of the least squares regression line for the given
dataset is y= 1.4x+ 4.
Question 4
Question
Suppose you are given a dataset with observations (x1, y1),(x2, y2), . . . , (xn, yn),
and you want to fit a linear regression model y=β0+β1xto the data. Find
the least squares estimates for the coefficients β0and β1.
Solution
Step 1: The linear regression model is given by:
y=β0+β1x
Step 2: The least squares estimates for β0and β1are obtained by minimizing
the sum of squared residuals, given by:
n
∑
i=1
(yi−(β0+β1xi))2
4
Step 3: To find the estimates, we need to minimize the sum of squared
residuals with respect to β0and β1simultaneously. Let Sbe the sum of squared
residuals.
Step 4: Taking the partial derivatives of Swith respect to β0and β1and
setting them equal to zero gives us the normal equations:
∂S
∂β0
=−2
n
∑
i=1
(yi−β0−β1xi) = 0
∂S
∂β1
=−2
n
∑
i=1
xi(yi−β0−β1xi) = 0
Step 5: Solving the normal equations simultaneously will give us the least
squares estimates for β0and β1. Let x=1
n∑n
i=1 xiand y=1
n∑n
i=1 yi.
Step 6: The estimates are given by:
ˆ
β1=∑n
i=1(xi−x)(yi−y)
∑n
i=1(xi−x)2
ˆ
β0=y−ˆ
β1x
Question 5
Question
Suppose we have a dataset with the following pairs of values:
x y
1 3
2 5
3 7
4 11
5 14
Calculate the least squares estimate of the linear regression model y=β0+
β1xfor this dataset.
Solution
Step 1: Calculate the means of xand y. The mean of xis ¯x=1+2+3+4+5
5= 3.
The mean of yis ¯y=3+5+7+11+14
5= 8.
Step 2: Calculate the deviations from the means. The deviations for xare:
x1= 1−3 = −2,x2= 2−3 = −1,x3= 3−3 = 0,x4= 4−3 = 1,x5= 5−3 = 2.
The deviations for yare: y1= 3−8 = −5,y2= 5−8 = −3,y3= 7−8 = −1,
y4= 11 −8 = 3,y5= 14 −8 = 6.
Step 3: Calculate the sum of products of deviations. ∑xiyi= (−2)(−5) +
(−1)(−3) + (0)(−1) + (1)(3) + (2)(6) = 4 + 3 + 0 + 3 + 12 = 22.
5
Step 4: Calculate the sum of squared deviations for x.∑x2
i= (−2)2+
(−1)2+ (0)2+ (1)2+ (2)2= 4 + 1 + 0 + 1 + 4 = 10.
Step 5: Calculate the least squares estimates ˆ
β1and ˆ
β0. Using the formulas:
ˆ
β1=∑xiyi
∑x2
i
=22
10 = 2.2,ˆ
β0= ¯y−ˆ
β1¯x= 8 −2.2×3 = 2.4.
Therefore, the least squares estimate of the linear regression model for this
dataset is ˆy= 2.4+2.2x.
Question 6
Question
Suppose we have a dataset with the following points: (1,3),(2,5),(3,7),(4,8),
(5,10). Find the least squares regression line for this dataset.
Solution
Step 1: Let’s first write down the formulas we need for finding the least squares
regression line. The equation of the least squares regression line is of the form
y=a+bx, where
b=n∑n
i=1 xiyi−∑n
i=1 xi∑n
i=1 yi
n∑n
i=1 x2
i−(∑n
i=1 xi)2
and
a=∑n
i=1 yi−b∑n
i=1 xi
n
Step 2: Calculate the necessary sums:
5
∑
i=1
xi= 15,
5
∑
i=1
yi= 33,
5
∑
i=1
x2
i= 55,
5
∑
i=1
xiyi= 143
Step 3: Substitute these values into the formulas for aand b:
b=5×143 −15 ×33
5×55 −152=715 −495
275 −225 =220
50 = 4.4
a=33 −4.4×15
5=33 −66
5=−6.6
Therefore, the least squares regression line for the given dataset is y=
−6.6+4.4x.
6
Question 7
Question
Let’s consider the following data points:
x y
1 4
2 7
3 11
4 12
5 16
Using linear regression, find the least squares estimation for the line that
best fits the data points.
Solution
Step 1: We start with the linear model:
y=β0+β1x
where β0is the intercept and β1is the slope of the line.
Step 2: We can rewrite the model in matrix form as:
Y=Xβ +ε
where
Y=
4
7
11
12
16
, X =
1 1
1 2
1 3
1 4
1 5
, β =[β0
β1], ε =
ε1
ε2
ε3
ε4
ε5
Step 3: The least squares estimates for βcan be obtained using the formula:
ˆ
β= (XTX)−1XTY
Step 4: Calculate XTX:
XTX=[11111
12345]
1 1
1 2
1 3
1 4
1 5
XTX=[5 15
15 55]
7
Step 5: Calculate (XTX)−1:
(XTX)−1=[5 15
15 55]−1
(XTX)−1=[11 −3
−3 1 ]
Step 6: Calculate XTY:
XTY=[11111
12345]
4
7
11
12
16
XTY=[50
190]
Step 7: Calculate ˆ
β:
ˆ
β=[11 −3
−3 1 ][50
190]
ˆ
β=[10
3]
Step 8: Therefore, the least squares estimation for the line that best fits the
data points is y= 10 + 3x.
Question 8
Question
Suppose we have a dataset with the following pairs of data points: (1,3),(2,5),(3,7),(4,10),
and (5,12). Determine the least squares estimate for the linear regression line
that best fits this dataset.
Solution
To find the least squares estimate for the linear regression line, we need to
minimize the sum of the squared differences between the observed y-values and
the predicted y-values. The regression line has the form y=mx +b, where m
is the slope and bis the intercept.
Step 1: Calculate the means of xand yFirst, find the means of xand
y:
¯x=1+2+3+4+5
5= 3
¯y=3+5+7+10+12
5= 7.4
8
Step 2: Calculate the slope mThe slope of the regression line can be
calculated using the formula:
m=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯
x)2
Substitute the given values:
m=(1 −3)(3 −7.4) + (2 −3)(5 −7.4) + (3 −3)(7 −7.4) + (4 −3)(10 −7.4) + (5 −3)(12 −7.4)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
=(−2)(−4.4) + (−1)(−2.4) + 0(0.6) + (1)(2.6) + (2)(4.6)
(−2)2+ (−1)2+ 02+ 12+ 22
=8.8+2.4+0+2.6+9.2
4+1+0+1+4
=22
10
= 2.2
Step 3: Calculate the intercept bUsing the formula for the intercept b:
b= ¯y−m¯x
Substitute the calculated values:
b= 7.4−2.2(3) = 7.4−6.6 = 0.8
Step 4: Write the regression line Therefore, the least squares estimate
for the linear regression line is:
ˆy= 2.2x+ 0.8
Question 9
Question
Suppose we have a dataset with five data points: (1,2),(2,3),(3,4),(4,6), and
(5,6). We want to fit a linear regression model Y=β0+β1Xto this data using
the method of least squares estimation. Calculate the least squares estimates
ˆ
β0and ˆ
β1.
Solution
Step 1: Write the model equation and the sum of squared errors (SSE). The
model equation is given by: Y=β0+β1X. Let ˆ
Yibe the predicted value of Y
for the ith observation. The sum of squared errors (SSE) is given by:
SSE =
n
∑
i=1
(Yi−ˆ
Yi)2
9
Step 2: Find the predicted values ˆ
Yi. For each data point (Xi, Yi), the
predicted value ˆ
Yiis given by:
ˆ
Yi=ˆ
β0+ˆ
β1Xi
Step 3: Calculate the residuals. The residual eifor each data point is defined
as:
ei=Yi−ˆ
Yi
Step 4: Set up the normal equations. The normal equations are given by:
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi) = 0
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)Xi= 0
Step 5: Solve the normal equations simultaneously. Substitute the data
points (1,2),(2,3),(3,4),(4,6), and (5,6) into the normal equations to obtain
two equations in two unknowns (ˆ
β0and ˆ
β1). Solve these equations to find the
least squares estimates ˆ
β0and ˆ
β1.
After solving these equations, you should find that ˆ
β0≈1.2and ˆ
β1≈0.9.
Question 11
Question
Suppose we have a dataset with the following pairs of observations: (1, 2), (2,
4), (3, 5), and (4, 7). Find the least squares estimation for the linear regression
model y=β0+β1x.
Solution
Step 1: We want to find the least squares estimation for the coefficients β0and
β1. The model is y=β0+β1x.
Step 2: We start by setting up the normal equations. For each observation
(xi, yi), the normal equations are:
n
∑
i=1
xi=β0n+β1
n
∑
i=1
xiyi
n
∑
i=1
xiyi=β0
n
∑
i=1
xi+β1
n
∑
i=1
x2
i
10
Step 3: Given the pairs of observations (1, 2), (2, 4), (3, 5), and (4, 7), we
have the following calculations:
4
∑
i=1
xi= 1 + 2 + 3 + 4 = 10
4
∑
i=1
yi= 2 + 4 + 5 + 7 = 18
4
∑
i=1
xiyi= (1 ·2) + (2 ·4) + (3 ·5) + (4 ·7) = 2 + 8 + 15 + 28 = 53
4
∑
i=1
x2
i= 12+ 22+ 32+ 42= 1 + 4 + 9 + 16 = 30
Step 4: Substituting the values into the normal equations, we get the fol-
lowing two equations:
10 = 4β0+ 53β1
53 = 10β0+ 30β1
Step 5: Solve the system of equations to find β0and β1.
Solving the system of equations, we find β0≈0.457 and β1≈0.807.
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis approximately y= 0.457 + 0.807x.
Question 12
Question
Suppose we have a dataset with the following values of xand y:
x y
1 3
2 5
3 7
4 9
Given that we want to perform linear regression to find the least squares
estimation of the regression coefficients β0and β1, what are the values of β0
and β1?
11
Solution
Step 1: Calculate the sample mean of x,¯x, and the sample mean of y,¯y. The
formulas for the sample means are:
¯x=1
n
n
∑
i=1
xiand ¯y=1
n
n
∑
i=1
yi
Calculating the sample means:
¯x=1
4(1 + 2 + 3 + 4) = 10
4= 2.5
¯y=1
4(3 + 5 + 7 + 9) = 24
4= 6
Step 2: Calculate the least squares estimators ˆ
β0and ˆ
β1. The formulas for
the estimators are:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
ˆ
β0= ¯y−ˆ
β1¯x
Calculating ˆ
β1:
ˆ
β1=(1 −2.5)(3 −6) + (2 −2.5)(5 −6) + (3 −2.5)(7 −6) + (4 −2.5)(9 −6)
(1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2
ˆ
β1=(−1)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
(−1.5)2+ (−0.5)2+ (0.5)2+ (1.5)2
ˆ
β1=3.5+0.5+0.5+4.5
2.25 + 0.25 + 0.25 + 2.25
ˆ
β1=9
5= 1.8
Calculating ˆ
β0:
ˆ
β0= 6 −1.8(2.5) = 6 −4.5 = 1.5
Therefore, the least squares estimators for the regression coefficients are
ˆ
β0= 1.5and ˆ
β1= 1.8.
Question 13
Question
Let Xbe a random variable with mean µXand variance σ2
X, and let Ybe
a random variable with mean µYand variance σ2
Y. Suppose the covariance
between Xand Yis given by Cov(X, Y ) = ρσXσYfor some constant ρ. Prove
that the slope of the least squares regression line of Yon Xis ρσY
σX.
12
Solution
Step 1: Recall that the slope of the least squares regression line of Yon Xis
given by
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 2: To prove this, we start by writing the equation of the least squares
regression line as
Y=ˆ
β0+ˆ
β1X
where ˆ
β0is the intercept and ˆ
β1is the slope.
Step 3: The least squares estimators ˆ
β0and ˆ
β1are chosen to minimize the
sum of squares of residuals, given by
SSRes =
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)2
Step 4: Using calculus, we find the estimates ˆ
β0and ˆ
β1by taking partial
derivatives with respect to ˆ
β0and ˆ
β1, setting them equal to zero, and solving
the resulting system of equations.
Step 5: Solving for ˆ
β1, we find
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 6: Thus, we have shown that the slope of the least squares regression
line of Yon Xis indeed ρσY
σX, as required.
Question 14
Question
Let Ybe a random variable that represents the sales of a certain product in a
given month, and let Xbe a random variable representing the amount spent
on advertising for the product in that same month. A sample of 8 data pairs
(xi, yi)is collected, yielding the following summary statistics:
∑xi= 80,∑yi= 400,∑x2
i= 800,∑y2
i= 3200,∑xiyi= 1800.
Using the method of least squares, determine the equation of the regression
line that best fits this data.
Solution
Step 1: The equation of the regression line is given by Y= ˆa+ˆ
bX, where ˆa
is the y-intercept and ˆ
bis the slope. In the method of least squares, we can
13
compute ˆ
busing the formula:
ˆ
b=∑(xiyi)−1
n(∑xi)(∑yi)
∑(x2
i)−1
n(∑xi)2
Step 2: Plugging in the given values:
ˆ
b=1800 −1
8(80)(400)
800 −1
8(80)2
Step 3: Calculating the numerator and denominator:
ˆ
b=1800 −4000
800 −640 =−2200
160 =−13.75
Step 4: Once we have ˆ
b, we can solve for ˆausing the formula:
ˆa= ¯y−ˆ
b¯x
Step 5: Using the formula ¯x=1
n∑xiand ¯y=1
n∑yi:
¯x=1
8(80) = 10,¯y=1
8(400) = 50
Step 6: Substituting the values into the formula to find ˆa:
ˆa= 50 −(−13.75)(10) = 187.5
Step 7: Therefore, the equation of the regression line that best fits the data
is Y= 187.5−13.75X.
Question 15
Question
Suppose we have a dataset with the following observations for variables Xand
Y:
X Y
1 3
2 5
3 7
4 10
Calculate the least squares estimate of the regression coefficients for the
linear model Y=β0+β1X.
14
Solution
Step 1: First, we need to calculate the mean values of Xand Y:
¯
X=1+2+3+4
4= 2.5and ¯
Y=3+5+7+10
4= 6.25
Step 2: Next, we calculate the sums of squares:
SSXX =
n
∑
i=1
(Xi−¯
X)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5.0
SSXY =
n
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (1−2.5)(3−6.25)+(2−2.5)(5−6.25)+(3−2.5)(7−6.25)+(4−2.5)(10−6.25) = 13.25
Step 3: Now we can calculate the least squares estimates for the regression
coefficients:
ˆ
β1=SSXY
SSXX
=13.25
5.0= 2.65
Step 4: Finally, we can use the formula ˆ
β0=¯
Y−ˆ
β1¯
Xto find:
ˆ
β0= 6.25 −2.65 ×2.5 = 6.25 −6.625 = −0.375
Therefore, the least squares estimate of the regression coefficients for the
linear model Y=β0+β1Xis ˆ
β0=−0.375 and ˆ
β1= 2.65.
Question 16
Question
Let’s consider a dataset with the following pairs of values:
x y
1 2
2 3
3 5
4 7
5 11
Determine the least squares estimation for the linear regression model y=
β0+β1x.
15
Solution
Step 1: Calculate the means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=2+3+5+7+11
5= 5.6
Step 2: Calculate the sum of the products of xand y:
n
∑
i=1
xiyi= 1 ·2+2·3+3·5+4·7+5·11 = 89
Step 3: Calculate the sum of the squares of x:
n
∑
i=1
x2
i= 12+ 22+ 32+ 42+ 52= 55
Step 4: Calculate the slope estimate ˆ
β1:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2=89 −5·3·5.6
55 −5·32=1.6
−10 =−0.16
Step 5: Calculate the intercept estimate ˆ
β0:
ˆ
β0= ¯y−ˆ
β1¯x= 5.6−(−0.16 ·3) = 6.08
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis ˆy= 6.08 −0.16x.
Question 17
Question
Suppose we have a set of data points given by (x1, y1),(x2, y2), . . . , (xn, yn),
where the relationship between the variables is modeled by a linear regression
line y=β0+β1x. Given that the least squares estimators for β0and β1are
denoted by ˆ
β0and ˆ
β1respectively, show that the covariance of these estimators
is equal to zero, i.e., Cov(ˆ
β0,ˆ
β1) = 0.
Solution
Step 1: To find the least squares estimators ˆ
β0and ˆ
β1, we need to minimize the
sum of squares of the vertical distances from the data points to the regression
line. Let ¯xbe the sample mean of the xvalues, ¯ybe the sample mean of the
yvalues, and sxx be the sample variance of the xvalues. The least squares
estimators are given by:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
16
ˆ
β0= ¯y−ˆ
β1¯x
Step 2: Now, let’s find the covariance of ˆ
β0and ˆ
β1. We know that the
covariance between two random variables Xand Yis defined as:
Cov(X, Y ) = E[(X−E[X])(Y−E[Y])]
Step 3: Calculate the covariance between ˆ
β0and ˆ
β1:
Cov(ˆ
β0,ˆ
β1) = E[( ˆ
β0−E[ˆ
β0])( ˆ
β1−E[ˆ
β1])]
Step 4: Since ˆ
β0and ˆ
β1are functions of the data points, we substitute their
expressions in terms of the data points:
Cov(ˆ
β0,ˆ
β1) = E[(¯y−ˆ
β1¯x−¯y)(ˆ
β1−E[ˆ
β1])]
Step 5: Simplify the expression and notice that since ˆ
β1is a function of the
data points, ˆ
β1is orthogonal to the sample mean of the xvalues. Therefore,
E[(¯x−¯x)·anything] = 0:
Cov(ˆ
β0,ˆ
β1) = Cov (¯y−ˆ
β1¯x, ˆ
β1)=−¯xCov(ˆ
β1,ˆ
β1)
Step 6: Finally, as the covariance of a random variable with itself is its
variance, we have:
Cov(ˆ
β0,ˆ
β1) = −¯xVar(ˆ
β1) = 0 ·Var(ˆ
β1) = 0
Therefore, we have shown that the covariance of the least squares estimators
ˆ
β0and ˆ
β1is equal to zero.
Question 18
Question
Suppose we have a dataset with the following pairs of observations:
x y
1 3
2 5
3 7
4 9
Find the least squares regression line for these data points.
17
Solution
Step 1: Calculate the mean of xand y. The mean of xis:
¯x=1+2+3+4
4= 2.5
The mean of yis:
¯y=3+5+7+9
4= 6
Step 2: Calculate the deviations from the mean for xand y, and the product
of these deviations. For x:
x x −¯x(x−¯x)2
1−1.5 2.25
2−0.5 0.25
3 0.5 0.25
4 1.5 2.25
For y:
y y −¯y(y−¯y)2
3−3 9
5−1 1
7 1 1
9 3 9
For the product of deviations:
x y x −¯x y −¯y
1 3 −1.5−3
2 5 −0.5−1
3 7 0.5 1
4 9 1.5 3
Step 3: Calculate the slope of the regression line. The slope bis given by:
b=∑(x−¯x)(y−¯y)
∑(x−¯x)2
Calculating the values gives:
b=(−1.5)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
2.25 + 0.25 + 0.25 + 2.25
=4.5+0.5+0.5+4.5
5.25
=10
5.25
≈1.90
18
Step 4: Calculate the intercept of the regression line. The intercept acan
be found using the equation:
a= ¯y−b¯x
Substitute the values to get:
a= 6 −1.90(2.5) = 1.75
Therefore, the least squares regression line for the data points is y= 1.90x+
1.75.
Question 19
Question
Let’s consider a dataset with the following values of X and Y:
X Y
1 2
2 3
3 5
4 7
5 11
Using linear regression, find the least squares estimation of the parameters
in the linear model Y=β0+β1X.
Solution
Step 1: Compute the means of X ( ¯
X) and Y ( ¯
Y):
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+11
5= 5.6
Step 2: Compute the deviations of X and Y from their respective means:
X X −¯
X Y −¯
Y
1−2−3.6
2−1−2.6
3 0 −0.6
4 1 1.4
5 2 5.4
Step 3: Compute the sum of the cross-deviations and the sum of the squares
of deviations of X:
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3.6)+(−1)(−2.6)+(0)(−0.6)+(1)(1.4)+(2)(5.4) = 26.6
19
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ (0)2+ (1)2+ (2)2= 10
Step 4: Compute the estimate for β1using the formula:
ˆ
β1=∑5
i=1(Xi−¯
X)(Yi−¯
Y)
∑5
i=1(Xi−¯
X)2=26.6
10 = 2.66
Step 5: Compute the estimate for β0using the formula:
ˆ
β0=¯
Y−ˆ
β1¯
X= 5.6−2.66 ×3 = 0.62
Therefore, the least squares estimation of the parameters in the linear model
Y=β0+β1Xare ˆ
β0= 0.62 and ˆ
β1= 2.66.
Question 20
Question
Let yi=β0+β1xi+ϵirepresent the simple linear regression model. Given
the following data points, use least squares estimation to find the regression
coefficients β0and β1:
(x1, y1) = (1,3),(x2, y2) = (2,5),(x3, y3) = (3,7),(x4, y4) = (4,9)
Solution
Step 1: Start by writing out the model for the given data points:
y1=β0+β1x1+ϵ1
y2=β0+β1x2+ϵ2
y3=β0+β1x3+ϵ3
y4=β0+β1x4+ϵ4
Step 2: We can rewrite the above equations using the given data points:
3 = β0+β1+ϵ1
5 = β0+ 2β1+ϵ2
7 = β0+ 3β1+ϵ3
9 = β0+ 4β1+ϵ4
20
Step 3: We want to minimize the sum of squared residuals, which are given
by ∑n
i=1 ϵ2
i. Subtitute the residuals ϵiwith yi−β0−β1xiin each equation:
e1= 3 −β0−β1=ϵ1
e2= 5 −β0−2β1=ϵ2
e3= 7 −β0−3β1=ϵ3
e4= 9 −β0−4β1=ϵ4
Step 4: Then, we can find the least squares estimators by differentiating the
sum of squares of residuals with respect to β0and β1, setting the result to 0,
and solving the resulting system of equations.
The solutions are: ˆ
β1= 2,ˆ
β0= 1
Therefore, the least squares estimates for the regression coefficients are ˆ
β0=
1and ˆ
β1= 2.
Question 21
Question
Suppose we have a dataset with 10 observations and two predictor variables x1
and x2. The model to be fitted is given by:
y=β0+β1x1+β2x2+ϵ
where ϵis the error term. The least squares estimates for the coefficients are as
follows: ˆ
β0= 3,ˆ
β1= 2,ˆ
β2=−1
The residual sum of squares is 20.
Determine the coefficient of determination (R2) for this model.
Solution
Step 1: Recall that the total sum of squares (TSS) can be calculated as:
TSS =
n
∑
i=1
(yi−¯y)2
where yiare the observed values of the dependent variable and ¯yis the mean of
the observed values.
Step 2: The residual sum of squares (RSS) is given as 20.
Step 3: The coefficient of determination R2is calculated as:
R2= 1 −RSS
TSS
21
Step 4: We need to calculate TSS. First, find the mean of the observed values
¯y:
¯y=1
n
n
∑
i=1
yi
Step 5: Next, calculate the total sum of squares (TSS):
TSS =
n
∑
i=1
(yi−¯y)2
Step 6: Finally, substitute the values of TSS, RSS, and calculate the coeffi-
cient of determination R2using the formula from Step 3.
Question 22
Question
Let’s consider the following dataset of pairs (xi, yi):
x y
1 3
2 5
3 7
4 11
5 13
Find the least squares estimate of the regression line y=β0+β1x.
Solution
Step 1: We need to find the sample means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=3+5+7+11+13
5= 7.8
Step 2: Compute the sums ∑xiand ∑yi:
∑xi= 1 + 2 + 3 + 4 + 5 = 15 and ∑yi= 3 + 5 + 7 + 11 + 13 = 39
Step 3: Compute the sums ∑x2
iand ∑xiyi:
∑x2
i= 12+22+32+42+52= 55 and ∑xiyi= 1·3+2·5+3·7+4·11+5·13 = 126
Step 4: Calculate the slope β1and the intercept β0using the formulas:
β1=∑(xi−¯x)(yi−¯y)
∑(xi−¯x)2and β0= ¯y−β1¯x
β1=126 −15 ·7.8
55 −15 ·3=126 −117
55 −45 =9
10 = 0.9
β0= 7.8−0.9·3 = 7.8−2.7 = 5.1
Therefore, the least squares estimate of the regression line is y= 5.1 + 0.9x.
22
Question 23
Question
Let’s consider a linear regression model with one independent variable:
Yi=β0+β1Xi+εi
Suppose we have the following data points:
(X1, Y1) = (1,3),(X2, Y2) = (2,5),(X3, Y3) = (3,8)
Determine the least squares estimates ˆ
β0and ˆ
β1for the coefficients.
Solution
Step 1: Set up the normal equations to find ˆ
β0and ˆ
β1. The normal equations
for the least squares estimates ˆ
β0and ˆ
β1are given by:
∑Yi=nˆ
β0+ˆ
β1∑Xi
∑XiYi=ˆ
β0∑Xi+ˆ
β1∑X2
i
Step 2: Calculate the required sums.
∑Xi= 1 + 2 + 3 = 6
∑Yi= 3 + 5 + 8 = 16
∑X2
i= 12+ 22+ 32= 14
∑XiYi= 1 ·3+2·5+3·8 = 3 + 10 + 24 = 37
Step 3: Solve the normal equations simultaneously. Substitute the values of
the sums into the normal equations:
16 = 3 ˆ
β0+ 6 ˆ
β1(1)
37 = 6 ˆ
β0+ 14 ˆ
β1(2)
Step 4: Solve the system of equations. Solving equations (1) and (2) simul-
taneously, we find:
ˆ
β1=16 ·14 −37 ·3
6·14 −6·6=186
60 = 3.1
ˆ
β0=16 −6·3.1
3=−0.7
Therefore, the least squares estimates are ˆ
β0=−0.7and ˆ
β1= 3.1.
23
Question 24
Question
Let Y=
2
4
5
6
and X=
1 1
1 2
1 3
1 4
. Find the least squares estimates of the coeffi-
cients b0and b1in the simple linear regression model Y=b0+b1X+ε.
Solution
Step 1: Calculate XTXand XTY.
XTX=[1111
1234]
1 1
1 2
1 3
1 4
=[4 10
10 30]
XTY=[1 1 1 1
1 2 3 4]
2
4
5
6
=[17
46]
Step 2: Calculate (XTX)−1.
(XTX)−1=1
(4)(30) −(10)(10) [30 −10
−10 4 ]=[3
8−1
4
−1
4
1
8]
Step 3: Calculate the least squares estimates b0and b1.
[b0
b1]= (XTX)−1XTY=[3
8−1
4
−1
4
1
8][17
46]=[3
7]
Therefore, the least squares estimates of the coefficients are b0= 3 and
b1= 7.
Question 25
Question
Suppose we have a dataset with the following observations for xand y:
x y
1 3
2 5
3 7
4 9
5 11
24
Find the least squares estimation of the regression line that best fits these
observations.
Solution
Step 1: Calculate the means of xand y: The mean of x(¯x) can be calculated
as:
¯x=1+2+3+4+5
5=15
5= 3
Similarly, the mean of y(¯y) can be calculated as:
¯y=3+5+7+9+11
5=35
5= 7
Step 2: Calculate the slope (β1) of the regression line: The formula for the
slope is:
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
Substitute the values into the formula:
β1=(1 −3)(3 −7) + (2 −3)(5 −7) + (3 −3)(7 −7) + (4 −3)(9 −7) + (5 −3)(11 −7)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
β1=(−2)(−4) + (−1)(−2) + (0)(0) + (1)(2) + (2)(4)
(−2)2+ (−1)2+ (0)2+ (1)2+ (2)2
β1=8+2+0+2+8
4+1+0+1+4 =20
10 = 2
Step 3: Calculate the intercept (β0) of the regression line: The formula for
the intercept is:
β0= ¯y−β1¯x
Substitute the values into the formula:
β0= 7 −2×3 = 7 −6 = 1
Therefore, the least squares estimation of the regression line is ˆy= 2x+ 1.
25
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+8
5= 5
Deviations from the mean:
X(X−¯
X) (Y−¯
Y)
1−2−3
2−1−2
3 0 0
4 1 2
5 2 3
Step 2: Calculate the sum of squared deviations from the mean of X(SXX ),
the sum of squared deviations from the mean of Y(SY Y ), and the sum of
cross-products of deviations (SXY ).
SXX =
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ 02+ 12+ 22= 10
SY Y =
5
∑
i=1
(Yi−¯
Y)2= (−3)2+ (−2)2+ 02+ 22+ 32= 22
SXY =
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3)+(−1)(−2)+(0)(0)+(1)(2)+(2)(3) = 13
Step 3: Calculate the slope of the least squares regression line:
b=SXY
SXX
=13
10 = 1.3
Step 4: Calculate the intercept of the least squares regression line:
a=¯
Y−b¯
X= 5 −1.3(3) = 0.1
Therefore, the equation of the least squares regression line is:
ˆ
Y= 0.1+1.3X
Question 2
Question
Let X=
1 1
1 2
1 3
1 4
be the design matrix and y=
2
1
3
5
be the response vector.
Find the least squares estimate ˆ
βof the regression coefficients in the model
Y=Xβ +ϵ, where ϵis the error term.
2
Solution
Step 1: The least squares estimate ˆ
βis given by the formula:
ˆ
β= (XTX)−1XTy.
Step 2: First, calculate XTX:
XTX=(1111
1234)
1 1
1 2
1 3
1 4
=(4 10
10 30).
Step 3: Next, calculate (XTX)−1:
(XTX)−1=(4 10
10 30)−1
=1
4·30 −10 ·10 (30 −10
−10 4 )=1
20 (3−1
−1 0.8).
Step 4: Now, calculate XTy:
XTy=(1111
1234)
2
1
3
5
=(11
26).
Step 5: Finally, calculate ˆ
β:
ˆ
β=1
20 (3−1
−1 0.8)(11
26)=1
20 (8
2)=(0.4
0.1).
Therefore, the least squares estimate of the regression coefficients is ˆ
β=
(0.4
0.1).
Question 3
Question
Suppose we have a dataset with the following points: (1, 5), (2, 7), (3, 9), (4,
11). We want to find the least squares regression line for this dataset. Find the
equation of the regression line.
Solution
Step 1: First, we need to calculate the mean of x (¯x) and the mean of y (¯y).
¯x=1+2+3+4
4= 2.5
3
¯y=5+7+9+11
4= 8
Step 2: Next, we calculate the sum of the squares of the deviations of x and
y from their means.
4
∑
i=1
(xi−¯x)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5
4
∑
i=1
(yi−¯y)2= (5 −8)2+ (7 −8)2+ (9 −8)2+ (11 −8)2= 10
Step 3: Now, we calculate the sum of the products of the deviations of x
and y from their means.
4
∑
i=1
(xi−¯x)(yi−¯y) = (1−2.5)(5−8)+(2−2.5)(7−8)+(3−2.5)(9−8)+(4−2.5)(11−8) = 7
Step 4: Using the formula for the slope of the least squares regression line,
we have:
b=∑4
i=1(xi−¯x)(yi−¯y)
∑4
i=1(xi−¯x)2=7
5= 1.4
Step 5: Finally, we find the y-intercept of the regression line using the for-
mula:
a= ¯y−b¯x= 8 −1.4(2.5) = 4
Therefore, the equation of the least squares regression line for the given
dataset is y= 1.4x+ 4.
Question 4
Question
Suppose you are given a dataset with observations (x1, y1),(x2, y2), . . . , (xn, yn),
and you want to fit a linear regression model y=β0+β1xto the data. Find
the least squares estimates for the coefficients β0and β1.
Solution
Step 1: The linear regression model is given by:
y=β0+β1x
Step 2: The least squares estimates for β0and β1are obtained by minimizing
the sum of squared residuals, given by:
n
∑
i=1
(yi−(β0+β1xi))2
4
Step 3: To find the estimates, we need to minimize the sum of squared
residuals with respect to β0and β1simultaneously. Let Sbe the sum of squared
residuals.
Step 4: Taking the partial derivatives of Swith respect to β0and β1and
setting them equal to zero gives us the normal equations:
∂S
∂β0
=−2
n
∑
i=1
(yi−β0−β1xi) = 0
∂S
∂β1
=−2
n
∑
i=1
xi(yi−β0−β1xi) = 0
Step 5: Solving the normal equations simultaneously will give us the least
squares estimates for β0and β1. Let x=1
n∑n
i=1 xiand y=1
n∑n
i=1 yi.
Step 6: The estimates are given by:
ˆ
β1=∑n
i=1(xi−x)(yi−y)
∑n
i=1(xi−x)2
ˆ
β0=y−ˆ
β1x
Question 5
Question
Suppose we have a dataset with the following pairs of values:
x y
1 3
2 5
3 7
4 11
5 14
Calculate the least squares estimate of the linear regression model y=β0+
β1xfor this dataset.
Solution
Step 1: Calculate the means of xand y. The mean of xis ¯x=1+2+3+4+5
5= 3.
The mean of yis ¯y=3+5+7+11+14
5= 8.
Step 2: Calculate the deviations from the means. The deviations for xare:
x1= 1−3 = −2,x2= 2−3 = −1,x3= 3−3 = 0,x4= 4−3 = 1,x5= 5−3 = 2.
The deviations for yare: y1= 3−8 = −5,y2= 5−8 = −3,y3= 7−8 = −1,
y4= 11 −8 = 3,y5= 14 −8 = 6.
Step 3: Calculate the sum of products of deviations. ∑xiyi= (−2)(−5) +
(−1)(−3) + (0)(−1) + (1)(3) + (2)(6) = 4 + 3 + 0 + 3 + 12 = 22.
5
Step 4: Calculate the sum of squared deviations for x.∑x2
i= (−2)2+
(−1)2+ (0)2+ (1)2+ (2)2= 4 + 1 + 0 + 1 + 4 = 10.
Step 5: Calculate the least squares estimates ˆ
β1and ˆ
β0. Using the formulas:
ˆ
β1=∑xiyi
∑x2
i
=22
10 = 2.2,ˆ
β0= ¯y−ˆ
β1¯x= 8 −2.2×3 = 2.4.
Therefore, the least squares estimate of the linear regression model for this
dataset is ˆy= 2.4+2.2x.
Question 6
Question
Suppose we have a dataset with the following points: (1,3),(2,5),(3,7),(4,8),
(5,10). Find the least squares regression line for this dataset.
Solution
Step 1: Let’s first write down the formulas we need for finding the least squares
regression line. The equation of the least squares regression line is of the form
y=a+bx, where
b=n∑n
i=1 xiyi−∑n
i=1 xi∑n
i=1 yi
n∑n
i=1 x2
i−(∑n
i=1 xi)2
and
a=∑n
i=1 yi−b∑n
i=1 xi
n
Step 2: Calculate the necessary sums:
5
∑
i=1
xi= 15,
5
∑
i=1
yi= 33,
5
∑
i=1
x2
i= 55,
5
∑
i=1
xiyi= 143
Step 3: Substitute these values into the formulas for aand b:
b=5×143 −15 ×33
5×55 −152=715 −495
275 −225 =220
50 = 4.4
a=33 −4.4×15
5=33 −66
5=−6.6
Therefore, the least squares regression line for the given dataset is y=
−6.6+4.4x.
6
Question 7
Question
Let’s consider the following data points:
x y
1 4
2 7
3 11
4 12
5 16
Using linear regression, find the least squares estimation for the line that
best fits the data points.
Solution
Step 1: We start with the linear model:
y=β0+β1x
where β0is the intercept and β1is the slope of the line.
Step 2: We can rewrite the model in matrix form as:
Y=Xβ +ε
where
Y=
4
7
11
12
16
, X =
1 1
1 2
1 3
1 4
1 5
, β =[β0
β1], ε =
ε1
ε2
ε3
ε4
ε5
Step 3: The least squares estimates for βcan be obtained using the formula:
ˆ
β= (XTX)−1XTY
Step 4: Calculate XTX:
XTX=[11111
12345]
1 1
1 2
1 3
1 4
1 5
XTX=[5 15
15 55]
7
Step 5: Calculate (XTX)−1:
(XTX)−1=[5 15
15 55]−1
(XTX)−1=[11 −3
−3 1 ]
Step 6: Calculate XTY:
XTY=[11111
12345]
4
7
11
12
16
XTY=[50
190]
Step 7: Calculate ˆ
β:
ˆ
β=[11 −3
−3 1 ][50
190]
ˆ
β=[10
3]
Step 8: Therefore, the least squares estimation for the line that best fits the
data points is y= 10 + 3x.
Question 8
Question
Suppose we have a dataset with the following pairs of data points: (1,3),(2,5),(3,7),(4,10),
and (5,12). Determine the least squares estimate for the linear regression line
that best fits this dataset.
Solution
To find the least squares estimate for the linear regression line, we need to
minimize the sum of the squared differences between the observed y-values and
the predicted y-values. The regression line has the form y=mx +b, where m
is the slope and bis the intercept.
Step 1: Calculate the means of xand yFirst, find the means of xand
y:
¯x=1+2+3+4+5
5= 3
¯y=3+5+7+10+12
5= 7.4
8
Step 2: Calculate the slope mThe slope of the regression line can be
calculated using the formula:
m=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯
x)2
Substitute the given values:
m=(1 −3)(3 −7.4) + (2 −3)(5 −7.4) + (3 −3)(7 −7.4) + (4 −3)(10 −7.4) + (5 −3)(12 −7.4)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
=(−2)(−4.4) + (−1)(−2.4) + 0(0.6) + (1)(2.6) + (2)(4.6)
(−2)2+ (−1)2+ 02+ 12+ 22
=8.8+2.4+0+2.6+9.2
4+1+0+1+4
=22
10
= 2.2
Step 3: Calculate the intercept bUsing the formula for the intercept b:
b= ¯y−m¯x
Substitute the calculated values:
b= 7.4−2.2(3) = 7.4−6.6 = 0.8
Step 4: Write the regression line Therefore, the least squares estimate
for the linear regression line is:
ˆy= 2.2x+ 0.8
Question 9
Question
Suppose we have a dataset with five data points: (1,2),(2,3),(3,4),(4,6), and
(5,6). We want to fit a linear regression model Y=β0+β1Xto this data using
the method of least squares estimation. Calculate the least squares estimates
ˆ
β0and ˆ
β1.
Solution
Step 1: Write the model equation and the sum of squared errors (SSE). The
model equation is given by: Y=β0+β1X. Let ˆ
Yibe the predicted value of Y
for the ith observation. The sum of squared errors (SSE) is given by:
SSE =
n
∑
i=1
(Yi−ˆ
Yi)2
9
Step 2: Find the predicted values ˆ
Yi. For each data point (Xi, Yi), the
predicted value ˆ
Yiis given by:
ˆ
Yi=ˆ
β0+ˆ
β1Xi
Step 3: Calculate the residuals. The residual eifor each data point is defined
as:
ei=Yi−ˆ
Yi
Step 4: Set up the normal equations. The normal equations are given by:
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi) = 0
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)Xi= 0
Step 5: Solve the normal equations simultaneously. Substitute the data
points (1,2),(2,3),(3,4),(4,6), and (5,6) into the normal equations to obtain
two equations in two unknowns (ˆ
β0and ˆ
β1). Solve these equations to find the
least squares estimates ˆ
β0and ˆ
β1.
After solving these equations, you should find that ˆ
β0≈1.2and ˆ
β1≈0.9.
Question 11
Question
Suppose we have a dataset with the following pairs of observations: (1, 2), (2,
4), (3, 5), and (4, 7). Find the least squares estimation for the linear regression
model y=β0+β1x.
Solution
Step 1: We want to find the least squares estimation for the coefficients β0and
β1. The model is y=β0+β1x.
Step 2: We start by setting up the normal equations. For each observation
(xi, yi), the normal equations are:
n
∑
i=1
xi=β0n+β1
n
∑
i=1
xiyi
n
∑
i=1
xiyi=β0
n
∑
i=1
xi+β1
n
∑
i=1
x2
i
10
Step 3: Given the pairs of observations (1, 2), (2, 4), (3, 5), and (4, 7), we
have the following calculations:
4
∑
i=1
xi= 1 + 2 + 3 + 4 = 10
4
∑
i=1
yi= 2 + 4 + 5 + 7 = 18
4
∑
i=1
xiyi= (1 ·2) + (2 ·4) + (3 ·5) + (4 ·7) = 2 + 8 + 15 + 28 = 53
4
∑
i=1
x2
i= 12+ 22+ 32+ 42= 1 + 4 + 9 + 16 = 30
Step 4: Substituting the values into the normal equations, we get the fol-
lowing two equations:
10 = 4β0+ 53β1
53 = 10β0+ 30β1
Step 5: Solve the system of equations to find β0and β1.
Solving the system of equations, we find β0≈0.457 and β1≈0.807.
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis approximately y= 0.457 + 0.807x.
Question 12
Question
Suppose we have a dataset with the following values of xand y:
x y
1 3
2 5
3 7
4 9
Given that we want to perform linear regression to find the least squares
estimation of the regression coefficients β0and β1, what are the values of β0
and β1?
11
Solution
Step 1: Calculate the sample mean of x,¯x, and the sample mean of y,¯y. The
formulas for the sample means are:
¯x=1
n
n
∑
i=1
xiand ¯y=1
n
n
∑
i=1
yi
Calculating the sample means:
¯x=1
4(1 + 2 + 3 + 4) = 10
4= 2.5
¯y=1
4(3 + 5 + 7 + 9) = 24
4= 6
Step 2: Calculate the least squares estimators ˆ
β0and ˆ
β1. The formulas for
the estimators are:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
ˆ
β0= ¯y−ˆ
β1¯x
Calculating ˆ
β1:
ˆ
β1=(1 −2.5)(3 −6) + (2 −2.5)(5 −6) + (3 −2.5)(7 −6) + (4 −2.5)(9 −6)
(1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2
ˆ
β1=(−1)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
(−1.5)2+ (−0.5)2+ (0.5)2+ (1.5)2
ˆ
β1=3.5+0.5+0.5+4.5
2.25 + 0.25 + 0.25 + 2.25
ˆ
β1=9
5= 1.8
Calculating ˆ
β0:
ˆ
β0= 6 −1.8(2.5) = 6 −4.5 = 1.5
Therefore, the least squares estimators for the regression coefficients are
ˆ
β0= 1.5and ˆ
β1= 1.8.
Question 13
Question
Let Xbe a random variable with mean µXand variance σ2
X, and let Ybe
a random variable with mean µYand variance σ2
Y. Suppose the covariance
between Xand Yis given by Cov(X, Y ) = ρσXσYfor some constant ρ. Prove
that the slope of the least squares regression line of Yon Xis ρσY
σX.
12
Solution
Step 1: Recall that the slope of the least squares regression line of Yon Xis
given by
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 2: To prove this, we start by writing the equation of the least squares
regression line as
Y=ˆ
β0+ˆ
β1X
where ˆ
β0is the intercept and ˆ
β1is the slope.
Step 3: The least squares estimators ˆ
β0and ˆ
β1are chosen to minimize the
sum of squares of residuals, given by
SSRes =
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)2
Step 4: Using calculus, we find the estimates ˆ
β0and ˆ
β1by taking partial
derivatives with respect to ˆ
β0and ˆ
β1, setting them equal to zero, and solving
the resulting system of equations.
Step 5: Solving for ˆ
β1, we find
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 6: Thus, we have shown that the slope of the least squares regression
line of Yon Xis indeed ρσY
σX, as required.
Question 14
Question
Let Ybe a random variable that represents the sales of a certain product in a
given month, and let Xbe a random variable representing the amount spent
on advertising for the product in that same month. A sample of 8 data pairs
(xi, yi)is collected, yielding the following summary statistics:
∑xi= 80,∑yi= 400,∑x2
i= 800,∑y2
i= 3200,∑xiyi= 1800.
Using the method of least squares, determine the equation of the regression
line that best fits this data.
Solution
Step 1: The equation of the regression line is given by Y= ˆa+ˆ
bX, where ˆa
is the y-intercept and ˆ
bis the slope. In the method of least squares, we can
13
compute ˆ
busing the formula:
ˆ
b=∑(xiyi)−1
n(∑xi)(∑yi)
∑(x2
i)−1
n(∑xi)2
Step 2: Plugging in the given values:
ˆ
b=1800 −1
8(80)(400)
800 −1
8(80)2
Step 3: Calculating the numerator and denominator:
ˆ
b=1800 −4000
800 −640 =−2200
160 =−13.75
Step 4: Once we have ˆ
b, we can solve for ˆausing the formula:
ˆa= ¯y−ˆ
b¯x
Step 5: Using the formula ¯x=1
n∑xiand ¯y=1
n∑yi:
¯x=1
8(80) = 10,¯y=1
8(400) = 50
Step 6: Substituting the values into the formula to find ˆa:
ˆa= 50 −(−13.75)(10) = 187.5
Step 7: Therefore, the equation of the regression line that best fits the data
is Y= 187.5−13.75X.
Question 15
Question
Suppose we have a dataset with the following observations for variables Xand
Y:
X Y
1 3
2 5
3 7
4 10
Calculate the least squares estimate of the regression coefficients for the
linear model Y=β0+β1X.
14
Solution
Step 1: First, we need to calculate the mean values of Xand Y:
¯
X=1+2+3+4
4= 2.5and ¯
Y=3+5+7+10
4= 6.25
Step 2: Next, we calculate the sums of squares:
SSXX =
n
∑
i=1
(Xi−¯
X)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5.0
SSXY =
n
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (1−2.5)(3−6.25)+(2−2.5)(5−6.25)+(3−2.5)(7−6.25)+(4−2.5)(10−6.25) = 13.25
Step 3: Now we can calculate the least squares estimates for the regression
coefficients:
ˆ
β1=SSXY
SSXX
=13.25
5.0= 2.65
Step 4: Finally, we can use the formula ˆ
β0=¯
Y−ˆ
β1¯
Xto find:
ˆ
β0= 6.25 −2.65 ×2.5 = 6.25 −6.625 = −0.375
Therefore, the least squares estimate of the regression coefficients for the
linear model Y=β0+β1Xis ˆ
β0=−0.375 and ˆ
β1= 2.65.
Question 16
Question
Let’s consider a dataset with the following pairs of values:
x y
1 2
2 3
3 5
4 7
5 11
Determine the least squares estimation for the linear regression model y=
β0+β1x.
15
Solution
Step 1: Calculate the means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=2+3+5+7+11
5= 5.6
Step 2: Calculate the sum of the products of xand y:
n
∑
i=1
xiyi= 1 ·2+2·3+3·5+4·7+5·11 = 89
Step 3: Calculate the sum of the squares of x:
n
∑
i=1
x2
i= 12+ 22+ 32+ 42+ 52= 55
Step 4: Calculate the slope estimate ˆ
β1:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2=89 −5·3·5.6
55 −5·32=1.6
−10 =−0.16
Step 5: Calculate the intercept estimate ˆ
β0:
ˆ
β0= ¯y−ˆ
β1¯x= 5.6−(−0.16 ·3) = 6.08
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis ˆy= 6.08 −0.16x.
Question 17
Question
Suppose we have a set of data points given by (x1, y1),(x2, y2), . . . , (xn, yn),
where the relationship between the variables is modeled by a linear regression
line y=β0+β1x. Given that the least squares estimators for β0and β1are
denoted by ˆ
β0and ˆ
β1respectively, show that the covariance of these estimators
is equal to zero, i.e., Cov(ˆ
β0,ˆ
β1) = 0.
Solution
Step 1: To find the least squares estimators ˆ
β0and ˆ
β1, we need to minimize the
sum of squares of the vertical distances from the data points to the regression
line. Let ¯xbe the sample mean of the xvalues, ¯ybe the sample mean of the
yvalues, and sxx be the sample variance of the xvalues. The least squares
estimators are given by:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
16
ˆ
β0= ¯y−ˆ
β1¯x
Step 2: Now, let’s find the covariance of ˆ
β0and ˆ
β1. We know that the
covariance between two random variables Xand Yis defined as:
Cov(X, Y ) = E[(X−E[X])(Y−E[Y])]
Step 3: Calculate the covariance between ˆ
β0and ˆ
β1:
Cov(ˆ
β0,ˆ
β1) = E[( ˆ
β0−E[ˆ
β0])( ˆ
β1−E[ˆ
β1])]
Step 4: Since ˆ
β0and ˆ
β1are functions of the data points, we substitute their
expressions in terms of the data points:
Cov(ˆ
β0,ˆ
β1) = E[(¯y−ˆ
β1¯x−¯y)(ˆ
β1−E[ˆ
β1])]
Step 5: Simplify the expression and notice that since ˆ
β1is a function of the
data points, ˆ
β1is orthogonal to the sample mean of the xvalues. Therefore,
E[(¯x−¯x)·anything] = 0:
Cov(ˆ
β0,ˆ
β1) = Cov (¯y−ˆ
β1¯x, ˆ
β1)=−¯xCov(ˆ
β1,ˆ
β1)
Step 6: Finally, as the covariance of a random variable with itself is its
variance, we have:
Cov(ˆ
β0,ˆ
β1) = −¯xVar(ˆ
β1) = 0 ·Var(ˆ
β1) = 0
Therefore, we have shown that the covariance of the least squares estimators
ˆ
β0and ˆ
β1is equal to zero.
Question 18
Question
Suppose we have a dataset with the following pairs of observations:
x y
1 3
2 5
3 7
4 9
Find the least squares regression line for these data points.
17
Solution
Step 1: Calculate the mean of xand y. The mean of xis:
¯x=1+2+3+4
4= 2.5
The mean of yis:
¯y=3+5+7+9
4= 6
Step 2: Calculate the deviations from the mean for xand y, and the product
of these deviations. For x:
x x −¯x(x−¯x)2
1−1.5 2.25
2−0.5 0.25
3 0.5 0.25
4 1.5 2.25
For y:
y y −¯y(y−¯y)2
3−3 9
5−1 1
7 1 1
9 3 9
For the product of deviations:
x y x −¯x y −¯y
1 3 −1.5−3
2 5 −0.5−1
3 7 0.5 1
4 9 1.5 3
Step 3: Calculate the slope of the regression line. The slope bis given by:
b=∑(x−¯x)(y−¯y)
∑(x−¯x)2
Calculating the values gives:
b=(−1.5)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
2.25 + 0.25 + 0.25 + 2.25
=4.5+0.5+0.5+4.5
5.25
=10
5.25
≈1.90
18
Step 4: Calculate the intercept of the regression line. The intercept acan
be found using the equation:
a= ¯y−b¯x
Substitute the values to get:
a= 6 −1.90(2.5) = 1.75
Therefore, the least squares regression line for the data points is y= 1.90x+
1.75.
Question 19
Question
Let’s consider a dataset with the following values of X and Y:
X Y
1 2
2 3
3 5
4 7
5 11
Using linear regression, find the least squares estimation of the parameters
in the linear model Y=β0+β1X.
Solution
Step 1: Compute the means of X ( ¯
X) and Y ( ¯
Y):
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+11
5= 5.6
Step 2: Compute the deviations of X and Y from their respective means:
X X −¯
X Y −¯
Y
1−2−3.6
2−1−2.6
3 0 −0.6
4 1 1.4
5 2 5.4
Step 3: Compute the sum of the cross-deviations and the sum of the squares
of deviations of X:
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3.6)+(−1)(−2.6)+(0)(−0.6)+(1)(1.4)+(2)(5.4) = 26.6
19
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ (0)2+ (1)2+ (2)2= 10
Step 4: Compute the estimate for β1using the formula:
ˆ
β1=∑5
i=1(Xi−¯
X)(Yi−¯
Y)
∑5
i=1(Xi−¯
X)2=26.6
10 = 2.66
Step 5: Compute the estimate for β0using the formula:
ˆ
β0=¯
Y−ˆ
β1¯
X= 5.6−2.66 ×3 = 0.62
Therefore, the least squares estimation of the parameters in the linear model
Y=β0+β1Xare ˆ
β0= 0.62 and ˆ
β1= 2.66.
Question 20
Question
Let yi=β0+β1xi+ϵirepresent the simple linear regression model. Given
the following data points, use least squares estimation to find the regression
coefficients β0and β1:
(x1, y1) = (1,3),(x2, y2) = (2,5),(x3, y3) = (3,7),(x4, y4) = (4,9)
Solution
Step 1: Start by writing out the model for the given data points:
y1=β0+β1x1+ϵ1
y2=β0+β1x2+ϵ2
y3=β0+β1x3+ϵ3
y4=β0+β1x4+ϵ4
Step 2: We can rewrite the above equations using the given data points:
3 = β0+β1+ϵ1
5 = β0+ 2β1+ϵ2
7 = β0+ 3β1+ϵ3
9 = β0+ 4β1+ϵ4
20
Step 3: We want to minimize the sum of squared residuals, which are given
by ∑n
i=1 ϵ2
i. Subtitute the residuals ϵiwith yi−β0−β1xiin each equation:
e1= 3 −β0−β1=ϵ1
e2= 5 −β0−2β1=ϵ2
e3= 7 −β0−3β1=ϵ3
e4= 9 −β0−4β1=ϵ4
Step 4: Then, we can find the least squares estimators by differentiating the
sum of squares of residuals with respect to β0and β1, setting the result to 0,
and solving the resulting system of equations.
The solutions are: ˆ
β1= 2,ˆ
β0= 1
Therefore, the least squares estimates for the regression coefficients are ˆ
β0=
1and ˆ
β1= 2.
Question 21
Question
Suppose we have a dataset with 10 observations and two predictor variables x1
and x2. The model to be fitted is given by:
y=β0+β1x1+β2x2+ϵ
where ϵis the error term. The least squares estimates for the coefficients are as
follows: ˆ
β0= 3,ˆ
β1= 2,ˆ
β2=−1
The residual sum of squares is 20.
Determine the coefficient of determination (R2) for this model.
Solution
Step 1: Recall that the total sum of squares (TSS) can be calculated as:
TSS =
n
∑
i=1
(yi−¯y)2
where yiare the observed values of the dependent variable and ¯yis the mean of
the observed values.
Step 2: The residual sum of squares (RSS) is given as 20.
Step 3: The coefficient of determination R2is calculated as:
R2= 1 −RSS
TSS
21
Step 4: We need to calculate TSS. First, find the mean of the observed values
¯y:
¯y=1
n
n
∑
i=1
yi
Step 5: Next, calculate the total sum of squares (TSS):
TSS =
n
∑
i=1
(yi−¯y)2
Step 6: Finally, substitute the values of TSS, RSS, and calculate the coeffi-
cient of determination R2using the formula from Step 3.
Question 22
Question
Let’s consider the following dataset of pairs (xi, yi):
x y
1 3
2 5
3 7
4 11
5 13
Find the least squares estimate of the regression line y=β0+β1x.
Solution
Step 1: We need to find the sample means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=3+5+7+11+13
5= 7.8
Step 2: Compute the sums ∑xiand ∑yi:
∑xi= 1 + 2 + 3 + 4 + 5 = 15 and ∑yi= 3 + 5 + 7 + 11 + 13 = 39
Step 3: Compute the sums ∑x2
iand ∑xiyi:
∑x2
i= 12+22+32+42+52= 55 and ∑xiyi= 1·3+2·5+3·7+4·11+5·13 = 126
Step 4: Calculate the slope β1and the intercept β0using the formulas:
β1=∑(xi−¯x)(yi−¯y)
∑(xi−¯x)2and β0= ¯y−β1¯x
β1=126 −15 ·7.8
55 −15 ·3=126 −117
55 −45 =9
10 = 0.9
β0= 7.8−0.9·3 = 7.8−2.7 = 5.1
Therefore, the least squares estimate of the regression line is y= 5.1 + 0.9x.
22
Question 23
Question
Let’s consider a linear regression model with one independent variable:
Yi=β0+β1Xi+εi
Suppose we have the following data points:
(X1, Y1) = (1,3),(X2, Y2) = (2,5),(X3, Y3) = (3,8)
Determine the least squares estimates ˆ
β0and ˆ
β1for the coefficients.
Solution
Step 1: Set up the normal equations to find ˆ
β0and ˆ
β1. The normal equations
for the least squares estimates ˆ
β0and ˆ
β1are given by:
∑Yi=nˆ
β0+ˆ
β1∑Xi
∑XiYi=ˆ
β0∑Xi+ˆ
β1∑X2
i
Step 2: Calculate the required sums.
∑Xi= 1 + 2 + 3 = 6
∑Yi= 3 + 5 + 8 = 16
∑X2
i= 12+ 22+ 32= 14
∑XiYi= 1 ·3+2·5+3·8 = 3 + 10 + 24 = 37
Step 3: Solve the normal equations simultaneously. Substitute the values of
the sums into the normal equations:
16 = 3 ˆ
β0+ 6 ˆ
β1(1)
37 = 6 ˆ
β0+ 14 ˆ
β1(2)
Step 4: Solve the system of equations. Solving equations (1) and (2) simul-
taneously, we find:
ˆ
β1=16 ·14 −37 ·3
6·14 −6·6=186
60 = 3.1
ˆ
β0=16 −6·3.1
3=−0.7
Therefore, the least squares estimates are ˆ
β0=−0.7and ˆ
β1= 3.1.
23
Question 24
Question
Let Y=
2
4
5
6
and X=
1 1
1 2
1 3
1 4
. Find the least squares estimates of the coeffi-
cients b0and b1in the simple linear regression model Y=b0+b1X+ε.
Solution
Step 1: Calculate XTXand XTY.
XTX=[1111
1234]
1 1
1 2
1 3
1 4
=[4 10
10 30]
XTY=[1 1 1 1
1 2 3 4]
2
4
5
6
=[17
46]
Step 2: Calculate (XTX)−1.
(XTX)−1=1
(4)(30) −(10)(10) [30 −10
−10 4 ]=[3
8−1
4
−1
4
1
8]
Step 3: Calculate the least squares estimates b0and b1.
[b0
b1]= (XTX)−1XTY=[3
8−1
4
−1
4
1
8][17
46]=[3
7]
Therefore, the least squares estimates of the coefficients are b0= 3 and
b1= 7.
Question 25
Question
Suppose we have a dataset with the following observations for xand y:
x y
1 3
2 5
3 7
4 9
5 11
24
Find the least squares estimation of the regression line that best fits these
observations.
Solution
Step 1: Calculate the means of xand y: The mean of x(¯x) can be calculated
as:
¯x=1+2+3+4+5
5=15
5= 3
Similarly, the mean of y(¯y) can be calculated as:
¯y=3+5+7+9+11
5=35
5= 7
Step 2: Calculate the slope (β1) of the regression line: The formula for the
slope is:
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
Substitute the values into the formula:
β1=(1 −3)(3 −7) + (2 −3)(5 −7) + (3 −3)(7 −7) + (4 −3)(9 −7) + (5 −3)(11 −7)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
β1=(−2)(−4) + (−1)(−2) + (0)(0) + (1)(2) + (2)(4)
(−2)2+ (−1)2+ (0)2+ (1)2+ (2)2
β1=8+2+0+2+8
4+1+0+1+4 =20
10 = 2
Step 3: Calculate the intercept (β0) of the regression line: The formula for
the intercept is:
β0= ¯y−β1¯x
Substitute the values into the formula:
β0= 7 −2×3 = 7 −6 = 1
Therefore, the least squares estimation of the regression line is ˆy= 2x+ 1.
25
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+8
5= 5
Deviations from the mean:
X(X−¯
X) (Y−¯
Y)
1−2−3
2−1−2
3 0 0
4 1 2
5 2 3
Step 2: Calculate the sum of squared deviations from the mean of X(SXX ),
the sum of squared deviations from the mean of Y(SY Y ), and the sum of
cross-products of deviations (SXY ).
SXX =
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ 02+ 12+ 22= 10
SY Y =
5
∑
i=1
(Yi−¯
Y)2= (−3)2+ (−2)2+ 02+ 22+ 32= 22
SXY =
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3)+(−1)(−2)+(0)(0)+(1)(2)+(2)(3) = 13
Step 3: Calculate the slope of the least squares regression line:
b=SXY
SXX
=13
10 = 1.3
Step 4: Calculate the intercept of the least squares regression line:
a=¯
Y−b¯
X= 5 −1.3(3) = 0.1
Therefore, the equation of the least squares regression line is:
ˆ
Y= 0.1+1.3X
Question 2
Question
Let X=
1 1
1 2
1 3
1 4
be the design matrix and y=
2
1
3
5
be the response vector.
Find the least squares estimate ˆ
βof the regression coefficients in the model
Y=Xβ +ϵ, where ϵis the error term.
2
Solution
Step 1: The least squares estimate ˆ
βis given by the formula:
ˆ
β= (XTX)−1XTy.
Step 2: First, calculate XTX:
XTX=(1111
1234)
1 1
1 2
1 3
1 4
=(4 10
10 30).
Step 3: Next, calculate (XTX)−1:
(XTX)−1=(4 10
10 30)−1
=1
4·30 −10 ·10 (30 −10
−10 4 )=1
20 (3−1
−1 0.8).
Step 4: Now, calculate XTy:
XTy=(1111
1234)
2
1
3
5
=(11
26).
Step 5: Finally, calculate ˆ
β:
ˆ
β=1
20 (3−1
−1 0.8)(11
26)=1
20 (8
2)=(0.4
0.1).
Therefore, the least squares estimate of the regression coefficients is ˆ
β=
(0.4
0.1).
Question 3
Question
Suppose we have a dataset with the following points: (1, 5), (2, 7), (3, 9), (4,
11). We want to find the least squares regression line for this dataset. Find the
equation of the regression line.
Solution
Step 1: First, we need to calculate the mean of x (¯x) and the mean of y (¯y).
¯x=1+2+3+4
4= 2.5
3
¯y=5+7+9+11
4= 8
Step 2: Next, we calculate the sum of the squares of the deviations of x and
y from their means.
4
∑
i=1
(xi−¯x)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5
4
∑
i=1
(yi−¯y)2= (5 −8)2+ (7 −8)2+ (9 −8)2+ (11 −8)2= 10
Step 3: Now, we calculate the sum of the products of the deviations of x
and y from their means.
4
∑
i=1
(xi−¯x)(yi−¯y) = (1−2.5)(5−8)+(2−2.5)(7−8)+(3−2.5)(9−8)+(4−2.5)(11−8) = 7
Step 4: Using the formula for the slope of the least squares regression line,
we have:
b=∑4
i=1(xi−¯x)(yi−¯y)
∑4
i=1(xi−¯x)2=7
5= 1.4
Step 5: Finally, we find the y-intercept of the regression line using the for-
mula:
a= ¯y−b¯x= 8 −1.4(2.5) = 4
Therefore, the equation of the least squares regression line for the given
dataset is y= 1.4x+ 4.
Question 4
Question
Suppose you are given a dataset with observations (x1, y1),(x2, y2), . . . , (xn, yn),
and you want to fit a linear regression model y=β0+β1xto the data. Find
the least squares estimates for the coefficients β0and β1.
Solution
Step 1: The linear regression model is given by:
y=β0+β1x
Step 2: The least squares estimates for β0and β1are obtained by minimizing
the sum of squared residuals, given by:
n
∑
i=1
(yi−(β0+β1xi))2
4
Step 3: To find the estimates, we need to minimize the sum of squared
residuals with respect to β0and β1simultaneously. Let Sbe the sum of squared
residuals.
Step 4: Taking the partial derivatives of Swith respect to β0and β1and
setting them equal to zero gives us the normal equations:
∂S
∂β0
=−2
n
∑
i=1
(yi−β0−β1xi) = 0
∂S
∂β1
=−2
n
∑
i=1
xi(yi−β0−β1xi) = 0
Step 5: Solving the normal equations simultaneously will give us the least
squares estimates for β0and β1. Let x=1
n∑n
i=1 xiand y=1
n∑n
i=1 yi.
Step 6: The estimates are given by:
ˆ
β1=∑n
i=1(xi−x)(yi−y)
∑n
i=1(xi−x)2
ˆ
β0=y−ˆ
β1x
Question 5
Question
Suppose we have a dataset with the following pairs of values:
x y
1 3
2 5
3 7
4 11
5 14
Calculate the least squares estimate of the linear regression model y=β0+
β1xfor this dataset.
Solution
Step 1: Calculate the means of xand y. The mean of xis ¯x=1+2+3+4+5
5= 3.
The mean of yis ¯y=3+5+7+11+14
5= 8.
Step 2: Calculate the deviations from the means. The deviations for xare:
x1= 1−3 = −2,x2= 2−3 = −1,x3= 3−3 = 0,x4= 4−3 = 1,x5= 5−3 = 2.
The deviations for yare: y1= 3−8 = −5,y2= 5−8 = −3,y3= 7−8 = −1,
y4= 11 −8 = 3,y5= 14 −8 = 6.
Step 3: Calculate the sum of products of deviations. ∑xiyi= (−2)(−5) +
(−1)(−3) + (0)(−1) + (1)(3) + (2)(6) = 4 + 3 + 0 + 3 + 12 = 22.
5
Step 4: Calculate the sum of squared deviations for x.∑x2
i= (−2)2+
(−1)2+ (0)2+ (1)2+ (2)2= 4 + 1 + 0 + 1 + 4 = 10.
Step 5: Calculate the least squares estimates ˆ
β1and ˆ
β0. Using the formulas:
ˆ
β1=∑xiyi
∑x2
i
=22
10 = 2.2,ˆ
β0= ¯y−ˆ
β1¯x= 8 −2.2×3 = 2.4.
Therefore, the least squares estimate of the linear regression model for this
dataset is ˆy= 2.4+2.2x.
Question 6
Question
Suppose we have a dataset with the following points: (1,3),(2,5),(3,7),(4,8),
(5,10). Find the least squares regression line for this dataset.
Solution
Step 1: Let’s first write down the formulas we need for finding the least squares
regression line. The equation of the least squares regression line is of the form
y=a+bx, where
b=n∑n
i=1 xiyi−∑n
i=1 xi∑n
i=1 yi
n∑n
i=1 x2
i−(∑n
i=1 xi)2
and
a=∑n
i=1 yi−b∑n
i=1 xi
n
Step 2: Calculate the necessary sums:
5
∑
i=1
xi= 15,
5
∑
i=1
yi= 33,
5
∑
i=1
x2
i= 55,
5
∑
i=1
xiyi= 143
Step 3: Substitute these values into the formulas for aand b:
b=5×143 −15 ×33
5×55 −152=715 −495
275 −225 =220
50 = 4.4
a=33 −4.4×15
5=33 −66
5=−6.6
Therefore, the least squares regression line for the given dataset is y=
−6.6+4.4x.
6
Question 7
Question
Let’s consider the following data points:
x y
1 4
2 7
3 11
4 12
5 16
Using linear regression, find the least squares estimation for the line that
best fits the data points.
Solution
Step 1: We start with the linear model:
y=β0+β1x
where β0is the intercept and β1is the slope of the line.
Step 2: We can rewrite the model in matrix form as:
Y=Xβ +ε
where
Y=
4
7
11
12
16
, X =
1 1
1 2
1 3
1 4
1 5
, β =[β0
β1], ε =
ε1
ε2
ε3
ε4
ε5
Step 3: The least squares estimates for βcan be obtained using the formula:
ˆ
β= (XTX)−1XTY
Step 4: Calculate XTX:
XTX=[11111
12345]
1 1
1 2
1 3
1 4
1 5
XTX=[5 15
15 55]
7
Step 5: Calculate (XTX)−1:
(XTX)−1=[5 15
15 55]−1
(XTX)−1=[11 −3
−3 1 ]
Step 6: Calculate XTY:
XTY=[11111
12345]
4
7
11
12
16
XTY=[50
190]
Step 7: Calculate ˆ
β:
ˆ
β=[11 −3
−3 1 ][50
190]
ˆ
β=[10
3]
Step 8: Therefore, the least squares estimation for the line that best fits the
data points is y= 10 + 3x.
Question 8
Question
Suppose we have a dataset with the following pairs of data points: (1,3),(2,5),(3,7),(4,10),
and (5,12). Determine the least squares estimate for the linear regression line
that best fits this dataset.
Solution
To find the least squares estimate for the linear regression line, we need to
minimize the sum of the squared differences between the observed y-values and
the predicted y-values. The regression line has the form y=mx +b, where m
is the slope and bis the intercept.
Step 1: Calculate the means of xand yFirst, find the means of xand
y:
¯x=1+2+3+4+5
5= 3
¯y=3+5+7+10+12
5= 7.4
8
Step 2: Calculate the slope mThe slope of the regression line can be
calculated using the formula:
m=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯
x)2
Substitute the given values:
m=(1 −3)(3 −7.4) + (2 −3)(5 −7.4) + (3 −3)(7 −7.4) + (4 −3)(10 −7.4) + (5 −3)(12 −7.4)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
=(−2)(−4.4) + (−1)(−2.4) + 0(0.6) + (1)(2.6) + (2)(4.6)
(−2)2+ (−1)2+ 02+ 12+ 22
=8.8+2.4+0+2.6+9.2
4+1+0+1+4
=22
10
= 2.2
Step 3: Calculate the intercept bUsing the formula for the intercept b:
b= ¯y−m¯x
Substitute the calculated values:
b= 7.4−2.2(3) = 7.4−6.6 = 0.8
Step 4: Write the regression line Therefore, the least squares estimate
for the linear regression line is:
ˆy= 2.2x+ 0.8
Question 9
Question
Suppose we have a dataset with five data points: (1,2),(2,3),(3,4),(4,6), and
(5,6). We want to fit a linear regression model Y=β0+β1Xto this data using
the method of least squares estimation. Calculate the least squares estimates
ˆ
β0and ˆ
β1.
Solution
Step 1: Write the model equation and the sum of squared errors (SSE). The
model equation is given by: Y=β0+β1X. Let ˆ
Yibe the predicted value of Y
for the ith observation. The sum of squared errors (SSE) is given by:
SSE =
n
∑
i=1
(Yi−ˆ
Yi)2
9
Step 2: Find the predicted values ˆ
Yi. For each data point (Xi, Yi), the
predicted value ˆ
Yiis given by:
ˆ
Yi=ˆ
β0+ˆ
β1Xi
Step 3: Calculate the residuals. The residual eifor each data point is defined
as:
ei=Yi−ˆ
Yi
Step 4: Set up the normal equations. The normal equations are given by:
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi) = 0
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)Xi= 0
Step 5: Solve the normal equations simultaneously. Substitute the data
points (1,2),(2,3),(3,4),(4,6), and (5,6) into the normal equations to obtain
two equations in two unknowns (ˆ
β0and ˆ
β1). Solve these equations to find the
least squares estimates ˆ
β0and ˆ
β1.
After solving these equations, you should find that ˆ
β0≈1.2and ˆ
β1≈0.9.
Question 11
Question
Suppose we have a dataset with the following pairs of observations: (1, 2), (2,
4), (3, 5), and (4, 7). Find the least squares estimation for the linear regression
model y=β0+β1x.
Solution
Step 1: We want to find the least squares estimation for the coefficients β0and
β1. The model is y=β0+β1x.
Step 2: We start by setting up the normal equations. For each observation
(xi, yi), the normal equations are:
n
∑
i=1
xi=β0n+β1
n
∑
i=1
xiyi
n
∑
i=1
xiyi=β0
n
∑
i=1
xi+β1
n
∑
i=1
x2
i
10
Step 3: Given the pairs of observations (1, 2), (2, 4), (3, 5), and (4, 7), we
have the following calculations:
4
∑
i=1
xi= 1 + 2 + 3 + 4 = 10
4
∑
i=1
yi= 2 + 4 + 5 + 7 = 18
4
∑
i=1
xiyi= (1 ·2) + (2 ·4) + (3 ·5) + (4 ·7) = 2 + 8 + 15 + 28 = 53
4
∑
i=1
x2
i= 12+ 22+ 32+ 42= 1 + 4 + 9 + 16 = 30
Step 4: Substituting the values into the normal equations, we get the fol-
lowing two equations:
10 = 4β0+ 53β1
53 = 10β0+ 30β1
Step 5: Solve the system of equations to find β0and β1.
Solving the system of equations, we find β0≈0.457 and β1≈0.807.
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis approximately y= 0.457 + 0.807x.
Question 12
Question
Suppose we have a dataset with the following values of xand y:
x y
1 3
2 5
3 7
4 9
Given that we want to perform linear regression to find the least squares
estimation of the regression coefficients β0and β1, what are the values of β0
and β1?
11
Solution
Step 1: Calculate the sample mean of x,¯x, and the sample mean of y,¯y. The
formulas for the sample means are:
¯x=1
n
n
∑
i=1
xiand ¯y=1
n
n
∑
i=1
yi
Calculating the sample means:
¯x=1
4(1 + 2 + 3 + 4) = 10
4= 2.5
¯y=1
4(3 + 5 + 7 + 9) = 24
4= 6
Step 2: Calculate the least squares estimators ˆ
β0and ˆ
β1. The formulas for
the estimators are:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
ˆ
β0= ¯y−ˆ
β1¯x
Calculating ˆ
β1:
ˆ
β1=(1 −2.5)(3 −6) + (2 −2.5)(5 −6) + (3 −2.5)(7 −6) + (4 −2.5)(9 −6)
(1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2
ˆ
β1=(−1)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
(−1.5)2+ (−0.5)2+ (0.5)2+ (1.5)2
ˆ
β1=3.5+0.5+0.5+4.5
2.25 + 0.25 + 0.25 + 2.25
ˆ
β1=9
5= 1.8
Calculating ˆ
β0:
ˆ
β0= 6 −1.8(2.5) = 6 −4.5 = 1.5
Therefore, the least squares estimators for the regression coefficients are
ˆ
β0= 1.5and ˆ
β1= 1.8.
Question 13
Question
Let Xbe a random variable with mean µXand variance σ2
X, and let Ybe
a random variable with mean µYand variance σ2
Y. Suppose the covariance
between Xand Yis given by Cov(X, Y ) = ρσXσYfor some constant ρ. Prove
that the slope of the least squares regression line of Yon Xis ρσY
σX.
12
Solution
Step 1: Recall that the slope of the least squares regression line of Yon Xis
given by
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 2: To prove this, we start by writing the equation of the least squares
regression line as
Y=ˆ
β0+ˆ
β1X
where ˆ
β0is the intercept and ˆ
β1is the slope.
Step 3: The least squares estimators ˆ
β0and ˆ
β1are chosen to minimize the
sum of squares of residuals, given by
SSRes =
n
∑
i=1
(Yi−ˆ
β0−ˆ
β1Xi)2
Step 4: Using calculus, we find the estimates ˆ
β0and ˆ
β1by taking partial
derivatives with respect to ˆ
β0and ˆ
β1, setting them equal to zero, and solving
the resulting system of equations.
Step 5: Solving for ˆ
β1, we find
ˆ
β1=Cov(X, Y )
V ar(X)=ρσXσY
σ2
X
=ρσY
σX
Step 6: Thus, we have shown that the slope of the least squares regression
line of Yon Xis indeed ρσY
σX, as required.
Question 14
Question
Let Ybe a random variable that represents the sales of a certain product in a
given month, and let Xbe a random variable representing the amount spent
on advertising for the product in that same month. A sample of 8 data pairs
(xi, yi)is collected, yielding the following summary statistics:
∑xi= 80,∑yi= 400,∑x2
i= 800,∑y2
i= 3200,∑xiyi= 1800.
Using the method of least squares, determine the equation of the regression
line that best fits this data.
Solution
Step 1: The equation of the regression line is given by Y= ˆa+ˆ
bX, where ˆa
is the y-intercept and ˆ
bis the slope. In the method of least squares, we can
13
compute ˆ
busing the formula:
ˆ
b=∑(xiyi)−1
n(∑xi)(∑yi)
∑(x2
i)−1
n(∑xi)2
Step 2: Plugging in the given values:
ˆ
b=1800 −1
8(80)(400)
800 −1
8(80)2
Step 3: Calculating the numerator and denominator:
ˆ
b=1800 −4000
800 −640 =−2200
160 =−13.75
Step 4: Once we have ˆ
b, we can solve for ˆausing the formula:
ˆa= ¯y−ˆ
b¯x
Step 5: Using the formula ¯x=1
n∑xiand ¯y=1
n∑yi:
¯x=1
8(80) = 10,¯y=1
8(400) = 50
Step 6: Substituting the values into the formula to find ˆa:
ˆa= 50 −(−13.75)(10) = 187.5
Step 7: Therefore, the equation of the regression line that best fits the data
is Y= 187.5−13.75X.
Question 15
Question
Suppose we have a dataset with the following observations for variables Xand
Y:
X Y
1 3
2 5
3 7
4 10
Calculate the least squares estimate of the regression coefficients for the
linear model Y=β0+β1X.
14
Solution
Step 1: First, we need to calculate the mean values of Xand Y:
¯
X=1+2+3+4
4= 2.5and ¯
Y=3+5+7+10
4= 6.25
Step 2: Next, we calculate the sums of squares:
SSXX =
n
∑
i=1
(Xi−¯
X)2= (1 −2.5)2+ (2 −2.5)2+ (3 −2.5)2+ (4 −2.5)2= 5.0
SSXY =
n
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (1−2.5)(3−6.25)+(2−2.5)(5−6.25)+(3−2.5)(7−6.25)+(4−2.5)(10−6.25) = 13.25
Step 3: Now we can calculate the least squares estimates for the regression
coefficients:
ˆ
β1=SSXY
SSXX
=13.25
5.0= 2.65
Step 4: Finally, we can use the formula ˆ
β0=¯
Y−ˆ
β1¯
Xto find:
ˆ
β0= 6.25 −2.65 ×2.5 = 6.25 −6.625 = −0.375
Therefore, the least squares estimate of the regression coefficients for the
linear model Y=β0+β1Xis ˆ
β0=−0.375 and ˆ
β1= 2.65.
Question 16
Question
Let’s consider a dataset with the following pairs of values:
x y
1 2
2 3
3 5
4 7
5 11
Determine the least squares estimation for the linear regression model y=
β0+β1x.
15
Solution
Step 1: Calculate the means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=2+3+5+7+11
5= 5.6
Step 2: Calculate the sum of the products of xand y:
n
∑
i=1
xiyi= 1 ·2+2·3+3·5+4·7+5·11 = 89
Step 3: Calculate the sum of the squares of x:
n
∑
i=1
x2
i= 12+ 22+ 32+ 42+ 52= 55
Step 4: Calculate the slope estimate ˆ
β1:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2=89 −5·3·5.6
55 −5·32=1.6
−10 =−0.16
Step 5: Calculate the intercept estimate ˆ
β0:
ˆ
β0= ¯y−ˆ
β1¯x= 5.6−(−0.16 ·3) = 6.08
Therefore, the least squares estimation for the linear regression model y=
β0+β1xis ˆy= 6.08 −0.16x.
Question 17
Question
Suppose we have a set of data points given by (x1, y1),(x2, y2), . . . , (xn, yn),
where the relationship between the variables is modeled by a linear regression
line y=β0+β1x. Given that the least squares estimators for β0and β1are
denoted by ˆ
β0and ˆ
β1respectively, show that the covariance of these estimators
is equal to zero, i.e., Cov(ˆ
β0,ˆ
β1) = 0.
Solution
Step 1: To find the least squares estimators ˆ
β0and ˆ
β1, we need to minimize the
sum of squares of the vertical distances from the data points to the regression
line. Let ¯xbe the sample mean of the xvalues, ¯ybe the sample mean of the
yvalues, and sxx be the sample variance of the xvalues. The least squares
estimators are given by:
ˆ
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
16
ˆ
β0= ¯y−ˆ
β1¯x
Step 2: Now, let’s find the covariance of ˆ
β0and ˆ
β1. We know that the
covariance between two random variables Xand Yis defined as:
Cov(X, Y ) = E[(X−E[X])(Y−E[Y])]
Step 3: Calculate the covariance between ˆ
β0and ˆ
β1:
Cov(ˆ
β0,ˆ
β1) = E[( ˆ
β0−E[ˆ
β0])( ˆ
β1−E[ˆ
β1])]
Step 4: Since ˆ
β0and ˆ
β1are functions of the data points, we substitute their
expressions in terms of the data points:
Cov(ˆ
β0,ˆ
β1) = E[(¯y−ˆ
β1¯x−¯y)(ˆ
β1−E[ˆ
β1])]
Step 5: Simplify the expression and notice that since ˆ
β1is a function of the
data points, ˆ
β1is orthogonal to the sample mean of the xvalues. Therefore,
E[(¯x−¯x)·anything] = 0:
Cov(ˆ
β0,ˆ
β1) = Cov (¯y−ˆ
β1¯x, ˆ
β1)=−¯xCov(ˆ
β1,ˆ
β1)
Step 6: Finally, as the covariance of a random variable with itself is its
variance, we have:
Cov(ˆ
β0,ˆ
β1) = −¯xVar(ˆ
β1) = 0 ·Var(ˆ
β1) = 0
Therefore, we have shown that the covariance of the least squares estimators
ˆ
β0and ˆ
β1is equal to zero.
Question 18
Question
Suppose we have a dataset with the following pairs of observations:
x y
1 3
2 5
3 7
4 9
Find the least squares regression line for these data points.
17
Solution
Step 1: Calculate the mean of xand y. The mean of xis:
¯x=1+2+3+4
4= 2.5
The mean of yis:
¯y=3+5+7+9
4= 6
Step 2: Calculate the deviations from the mean for xand y, and the product
of these deviations. For x:
x x −¯x(x−¯x)2
1−1.5 2.25
2−0.5 0.25
3 0.5 0.25
4 1.5 2.25
For y:
y y −¯y(y−¯y)2
3−3 9
5−1 1
7 1 1
9 3 9
For the product of deviations:
x y x −¯x y −¯y
1 3 −1.5−3
2 5 −0.5−1
3 7 0.5 1
4 9 1.5 3
Step 3: Calculate the slope of the regression line. The slope bis given by:
b=∑(x−¯x)(y−¯y)
∑(x−¯x)2
Calculating the values gives:
b=(−1.5)(−3) + (−0.5)(−1) + (0.5)(1) + (1.5)(3)
2.25 + 0.25 + 0.25 + 2.25
=4.5+0.5+0.5+4.5
5.25
=10
5.25
≈1.90
18
Step 4: Calculate the intercept of the regression line. The intercept acan
be found using the equation:
a= ¯y−b¯x
Substitute the values to get:
a= 6 −1.90(2.5) = 1.75
Therefore, the least squares regression line for the data points is y= 1.90x+
1.75.
Question 19
Question
Let’s consider a dataset with the following values of X and Y:
X Y
1 2
2 3
3 5
4 7
5 11
Using linear regression, find the least squares estimation of the parameters
in the linear model Y=β0+β1X.
Solution
Step 1: Compute the means of X ( ¯
X) and Y ( ¯
Y):
¯
X=1+2+3+4+5
5= 3 and ¯
Y=2+3+5+7+11
5= 5.6
Step 2: Compute the deviations of X and Y from their respective means:
X X −¯
X Y −¯
Y
1−2−3.6
2−1−2.6
3 0 −0.6
4 1 1.4
5 2 5.4
Step 3: Compute the sum of the cross-deviations and the sum of the squares
of deviations of X:
5
∑
i=1
(Xi−¯
X)(Yi−¯
Y) = (−2)(−3.6)+(−1)(−2.6)+(0)(−0.6)+(1)(1.4)+(2)(5.4) = 26.6
19
5
∑
i=1
(Xi−¯
X)2= (−2)2+ (−1)2+ (0)2+ (1)2+ (2)2= 10
Step 4: Compute the estimate for β1using the formula:
ˆ
β1=∑5
i=1(Xi−¯
X)(Yi−¯
Y)
∑5
i=1(Xi−¯
X)2=26.6
10 = 2.66
Step 5: Compute the estimate for β0using the formula:
ˆ
β0=¯
Y−ˆ
β1¯
X= 5.6−2.66 ×3 = 0.62
Therefore, the least squares estimation of the parameters in the linear model
Y=β0+β1Xare ˆ
β0= 0.62 and ˆ
β1= 2.66.
Question 20
Question
Let yi=β0+β1xi+ϵirepresent the simple linear regression model. Given
the following data points, use least squares estimation to find the regression
coefficients β0and β1:
(x1, y1) = (1,3),(x2, y2) = (2,5),(x3, y3) = (3,7),(x4, y4) = (4,9)
Solution
Step 1: Start by writing out the model for the given data points:
y1=β0+β1x1+ϵ1
y2=β0+β1x2+ϵ2
y3=β0+β1x3+ϵ3
y4=β0+β1x4+ϵ4
Step 2: We can rewrite the above equations using the given data points:
3 = β0+β1+ϵ1
5 = β0+ 2β1+ϵ2
7 = β0+ 3β1+ϵ3
9 = β0+ 4β1+ϵ4
20
Step 3: We want to minimize the sum of squared residuals, which are given
by ∑n
i=1 ϵ2
i. Subtitute the residuals ϵiwith yi−β0−β1xiin each equation:
e1= 3 −β0−β1=ϵ1
e2= 5 −β0−2β1=ϵ2
e3= 7 −β0−3β1=ϵ3
e4= 9 −β0−4β1=ϵ4
Step 4: Then, we can find the least squares estimators by differentiating the
sum of squares of residuals with respect to β0and β1, setting the result to 0,
and solving the resulting system of equations.
The solutions are: ˆ
β1= 2,ˆ
β0= 1
Therefore, the least squares estimates for the regression coefficients are ˆ
β0=
1and ˆ
β1= 2.
Question 21
Question
Suppose we have a dataset with 10 observations and two predictor variables x1
and x2. The model to be fitted is given by:
y=β0+β1x1+β2x2+ϵ
where ϵis the error term. The least squares estimates for the coefficients are as
follows: ˆ
β0= 3,ˆ
β1= 2,ˆ
β2=−1
The residual sum of squares is 20.
Determine the coefficient of determination (R2) for this model.
Solution
Step 1: Recall that the total sum of squares (TSS) can be calculated as:
TSS =
n
∑
i=1
(yi−¯y)2
where yiare the observed values of the dependent variable and ¯yis the mean of
the observed values.
Step 2: The residual sum of squares (RSS) is given as 20.
Step 3: The coefficient of determination R2is calculated as:
R2= 1 −RSS
TSS
21
Step 4: We need to calculate TSS. First, find the mean of the observed values
¯y:
¯y=1
n
n
∑
i=1
yi
Step 5: Next, calculate the total sum of squares (TSS):
TSS =
n
∑
i=1
(yi−¯y)2
Step 6: Finally, substitute the values of TSS, RSS, and calculate the coeffi-
cient of determination R2using the formula from Step 3.
Question 22
Question
Let’s consider the following dataset of pairs (xi, yi):
x y
1 3
2 5
3 7
4 11
5 13
Find the least squares estimate of the regression line y=β0+β1x.
Solution
Step 1: We need to find the sample means of xand y:
¯x=1+2+3+4+5
5= 3 and ¯y=3+5+7+11+13
5= 7.8
Step 2: Compute the sums ∑xiand ∑yi:
∑xi= 1 + 2 + 3 + 4 + 5 = 15 and ∑yi= 3 + 5 + 7 + 11 + 13 = 39
Step 3: Compute the sums ∑x2
iand ∑xiyi:
∑x2
i= 12+22+32+42+52= 55 and ∑xiyi= 1·3+2·5+3·7+4·11+5·13 = 126
Step 4: Calculate the slope β1and the intercept β0using the formulas:
β1=∑(xi−¯x)(yi−¯y)
∑(xi−¯x)2and β0= ¯y−β1¯x
β1=126 −15 ·7.8
55 −15 ·3=126 −117
55 −45 =9
10 = 0.9
β0= 7.8−0.9·3 = 7.8−2.7 = 5.1
Therefore, the least squares estimate of the regression line is y= 5.1 + 0.9x.
22
Question 23
Question
Let’s consider a linear regression model with one independent variable:
Yi=β0+β1Xi+εi
Suppose we have the following data points:
(X1, Y1) = (1,3),(X2, Y2) = (2,5),(X3, Y3) = (3,8)
Determine the least squares estimates ˆ
β0and ˆ
β1for the coefficients.
Solution
Step 1: Set up the normal equations to find ˆ
β0and ˆ
β1. The normal equations
for the least squares estimates ˆ
β0and ˆ
β1are given by:
∑Yi=nˆ
β0+ˆ
β1∑Xi
∑XiYi=ˆ
β0∑Xi+ˆ
β1∑X2
i
Step 2: Calculate the required sums.
∑Xi= 1 + 2 + 3 = 6
∑Yi= 3 + 5 + 8 = 16
∑X2
i= 12+ 22+ 32= 14
∑XiYi= 1 ·3+2·5+3·8 = 3 + 10 + 24 = 37
Step 3: Solve the normal equations simultaneously. Substitute the values of
the sums into the normal equations:
16 = 3 ˆ
β0+ 6 ˆ
β1(1)
37 = 6 ˆ
β0+ 14 ˆ
β1(2)
Step 4: Solve the system of equations. Solving equations (1) and (2) simul-
taneously, we find:
ˆ
β1=16 ·14 −37 ·3
6·14 −6·6=186
60 = 3.1
ˆ
β0=16 −6·3.1
3=−0.7
Therefore, the least squares estimates are ˆ
β0=−0.7and ˆ
β1= 3.1.
23
Question 24
Question
Let Y=
2
4
5
6
and X=
1 1
1 2
1 3
1 4
. Find the least squares estimates of the coeffi-
cients b0and b1in the simple linear regression model Y=b0+b1X+ε.
Solution
Step 1: Calculate XTXand XTY.
XTX=[1111
1234]
1 1
1 2
1 3
1 4
=[4 10
10 30]
XTY=[1 1 1 1
1 2 3 4]
2
4
5
6
=[17
46]
Step 2: Calculate (XTX)−1.
(XTX)−1=1
(4)(30) −(10)(10) [30 −10
−10 4 ]=[3
8−1
4
−1
4
1
8]
Step 3: Calculate the least squares estimates b0and b1.
[b0
b1]= (XTX)−1XTY=[3
8−1
4
−1
4
1
8][17
46]=[3
7]
Therefore, the least squares estimates of the coefficients are b0= 3 and
b1= 7.
Question 25
Question
Suppose we have a dataset with the following observations for xand y:
x y
1 3
2 5
3 7
4 9
5 11
24
Find the least squares estimation of the regression line that best fits these
observations.
Solution
Step 1: Calculate the means of xand y: The mean of x(¯x) can be calculated
as:
¯x=1+2+3+4+5
5=15
5= 3
Similarly, the mean of y(¯y) can be calculated as:
¯y=3+5+7+9+11
5=35
5= 7
Step 2: Calculate the slope (β1) of the regression line: The formula for the
slope is:
β1=∑n
i=1(xi−¯x)(yi−¯y)
∑n
i=1(xi−¯x)2
Substitute the values into the formula:
β1=(1 −3)(3 −7) + (2 −3)(5 −7) + (3 −3)(7 −7) + (4 −3)(9 −7) + (5 −3)(11 −7)
(1 −3)2+ (2 −3)2+ (3 −3)2+ (4 −3)2+ (5 −3)2
β1=(−2)(−4) + (−1)(−2) + (0)(0) + (1)(2) + (2)(4)
(−2)2+ (−1)2+ (0)2+ (1)2+ (2)2
β1=8+2+0+2+8
4+1+0+1+4 =20
10 = 2
Step 3: Calculate the intercept (β0) of the regression line: The formula for
the intercept is:
β0= ¯y−β1¯x
Substitute the values into the formula:
β0= 7 −2×3 = 7 −6 = 1
Therefore, the least squares estimation of the regression line is ˆy= 2x+ 1.
25
