DUE IN 6 HOURS!!!! Linear algebra, matlab knowledge required, handshake required

Goals:

1. Introduce the concept of a least squares regression in MATLAB.

2. Apply least squares for linear regression against a data set.

3. Apply least squares for polynomial curve fitting against a data set.

Introduction

A common task in science and engineering is to model and analyze relationships existing in the natural world. As an example, consider the relationship between running speed and an individual’s heart rate. If we could quantize this phenomenon within the context of a mathematical model, then we could stand to make more detailed conclusions of the true relationship that exists.

In the simplest terms, we can cast this problem as modeling via some mathematical equation having a standard form (i.e. linear, exponential, polynomial, logarithmic etc.). Once we have decided upon the standard form of the model, we can then collect data and come up with an estimation of the parameters for the model we chose. Finally, we can see how closely the data points predicted by our model lines up with the actual data points found.

To do this parameter estimation, we will use least squares estimation . This topic will be covered in more detail in later lectures. Here we introduce the minimum needed for implementation.

Derivation of Least Squares Estimation: Linear Model

The simplest model between an independent variable x and a dependent variable y, is a straight line. Every linear function is uniquely specified by just two parameters, its slope and the intercept or offset along the dependent axis.

y

rise

run

x

Parameters for a Linear Model

Any two distinct points, uniquely determine a linear function, as long as they have different

x-values. Below, we derive the equation for the line through two distinct points, in a way that will hopefully elucidate the least square calculations to appear a bit later.

The figure shows graphically the unique linear function through two given points and.

x

y

Two Points Determine a Line

Given two points, how can we find the line that goes through both of them? Well, the line must have the form:

Since both points lie on this line, we have two equations for the two unknown parameters and .

Point 1:

Point 2:

Now define the parameter vector and the observation vector y.

and introduce the design matrix:

Notice the design matrix only depends on the x-coordinates of the points, which form its second column. The y-coordinates only appear in the observation vector, and do not appear in D at all.

Using these definitions, we can write both equations for the two points in vector/matrix form:

The determinant of the design matrix D is non-zero, provided the two points have different

x-values.

Recall a vertical line does not define a linear function! (It fails the vertical line test!)

Provided the two points are distinct and not directly above one another, there is always a unique solution. The design matrix has the inverse:

and the unique solution is:

You should recognize this result as slope equals

rise over run.

Thus, the line through these two points has the intercept and slope:

An Alternative Derivation

Here is another derivation of the same problem with two points determining a line. It looks way more complicated, but the advantage is, it can be generalized to any number of points – not just two! Let's define the matrix product:

This matrix is invertible, (unless ), since its determinant is:

Its inverse is:

Now we are ready to resolve the equation:

First multiply both sides by obtaining:

Now multiply by and exchange sides to find:

If you expand out the product on the RHS, you will find (after some cancellations), that this gives the exact same result for the slope and intercept that we found earlier. Further, this new equation is valid no matter how many points we need to fit to our linear model.

An Arbitrary Number of Points

Let us now consider some experimental data for an athlete's heart rate as a function of their running speed. We can write in general for any given running speed, xi , and heart rate, yi :