calculus problem

G5205 Intro to Linear Regression Lecture 1 - 9/12/2016

Review of Calculus Lecturer: Arian Maleki Scribe: Arian Maleki

One of the main tools used in this course is calculus. Concepts like derivative, local maxima and minima, second derivative, mean value theorem, gradient, and Hessian will be used multiple times in this course. We briefly review these topics in this lecture. I suggest to those of you who are interested in the “history of science”to read about the history of calculus. While Newton and Leibniz developed the concepts of infinite small quantities etc. and somehow developed the notion of derivative and even Taylor expansion, they were not accepted by many mathematicians as mathematically “rigorous enough” tools. It took people a while to give a mathematically-satisfying meaning to derivatives and stuff. These were considered as major breakthroughs four hundred years ago.

1 Uni-variable calculus

In this section we study the properties of univarible functions, i.e., f : R → R. The graph of this function is shown in Figure 1. There are many questions we can ask about this function. One of the most important questions is to find locations of the maximas or minimas. The derivative of f is a tool that helps us answer this question more easily. Before we mention what the derivative is, we formally define the concept of a local minima or maxima.

Definition 1. The point x ∈ R is a local maxima of f if and only if there exists a neighborhood of x, let’s call it (x− �,x + �), where � > 0 such that ∀z ∈ (x− �,x + �) we have

f(z) ≤ f(x).

Similarly the concept of local maxima can be defined. Can you explain how? Can you find the local max- imas and minimas in Figure 1. One of the main objectives of calculus is to find the local minimas(maximas) of a function. The derivative of a function is a tool that often simplifies this question. Below we first review the concept of derivative.

Definition 2. The derivative of a function f : R → R at a point x is defined as

f′(x) = lim ∆x→0

f(x + ∆x) −f(x) ∆x

.

Note that the derivative is defined only if the above limit exists. If the limit does not exist then the function is not differentiable at x and we cannot define the derivative. In this course we see non-differentiable functions in a few occasions, but most of the functions we will deal with are differentiable for every x.

Please check Figure 1. Can you use this figure to (intuitively) confirm that if we draw a line tangent to the diagram of f at point x, then its slope is the derivative of a function at that point? Then convince yourself that if x is a local minima or a local maxima of f, f′(x) = 0. This is in fact one of the most important results you learned in calculus.

Theorem 3. Suppose that x∗ is a local minima (or maxima) of f : R → R. If f is differentiable at x∗, then f′(x∗) = 0.

Since most of you have seen and used this theorem many times already we do not spend more time on it. Another important theorem that you learned in univariate calculus and is used very often (specially when you want to prove things) is the mean value theorem.

1

�x

f (x + �x) � f (x)

x

Figure 1: Diagram of a univariate function. From this figure you can see that f(x+∆x)−f(x)

∆x for a given ∆x is an

estimate of the slope of the line that is drawn in black. Hence, as ∆x shrinks it becomes the slope of the tangent line at point x.

Theorem 4 (Mean value theorem). Let x < y and suppose that f : R → R is differentiable on every point in the interval [x,y]. Then there exists a point z ∈ [x,y] such that

f(y) −f(x) = f′(z)(y −x).

Can you explain on a graph what this theorem means? We will see a couple of applications of this theorem later in this lecture.

2 Multivariable calculus

In this course, often we would like to find the minimas and maximas of function of more than one variable. In other words, we have a function f : Rp → R, and we would like to know where it takes its maximum and minimum. Here is a simple example of univariable function:

Example 1 f : R2 → R with f(x1,x2) = (x1 − 1)2 + (x2 − 1)2 + x1x2. For instance, f(1, 1) = 1 and f(0, 0) = 2.

Remark Let me go over the terminology that I will use in the rest of this lecture. The boldfaced letters such as x denote vectors. Then the elements of the vector are denoted with x1,x2, . . .. Hence, x = (x1,x2, . . . ,xp)

T .

Similar to the univariable case, we would like to use derivatives to simplify the problem of finding the minimum of f. Before I start our discussion let me tell you about the notation we will use. For vectors we will use boldfaced letters such as x. The elements of the vector x will then be denoted by x1,x2, . . ..

To gain some intuition on the conditions for the local minimas and maximas of f(x), let’s consider a function of two variables x1 and x2. Suppose that x̄ denotes the minimum of this function. How is the minimum defined? Similar to the univariate case. Except that the neighborhood of the point x̄ ∈ Rp is

2

defined as

NR(x̄) , {z ∈ Rp | ‖z − x̄‖22 = p∑ i=1

(zi −xi)2 < R2}. (1)

Explain what the shape of this neighborhood is when p = 3? As I mentioned before our goal is to use derivatives to find the local maximizers and minimizers of a

function. Let me ask you a simpler question. Suppose that in Figure 2 x̄ = (x̄1, x̄2) is the minimizer of f. Also, suppose that an oracle has given you that x̄1 = 0.5, and asks you the value of x̄2. How would you respond? Here is how I answer. First note that since x̄ is a minimizer, then if we fix x̄1 and consider the uni-variable function:

g(x2) = f(0.5,x2),

then this function takes its minimum at x2 = x̄2. Can you prove this claim or explain why it is true? If this is the case, then we can say that g′(x̄2) is zero. The derivative of g(x2) is called partial derivative of f with respect to x2 and is denoted with

∂f ∂x2

(x1,x2). It can also be defined in the following way:

∂f

∂x2 (x̄1, x̄2) = lim

∆x2→0

f(x̄1, x̄2 + ∆x2) −f(x̄1, x̄2) ∆x2

.

Our argument has proved that if x̄ is the minimum of f, then

∂f

∂x2 (x̄) = 0.

Similarly, we can argue that ∂f

∂x1 (x̄) = 0.

Is it clear why these equations are correct? Now check Figure 2. So far we have argued that since x̄ is the minimum of the function, then if we look at the graph of the function in the direction of a or b, then x̄ should be a minimum of that graph and hence ∂f

∂x1 (x̄) = 0 and ∂f

∂x2 (x̄) = 0. It seems that if we look at the

graph of the function in an arbitrary direction, let’s say c (See Fig 2), then again that graph should take its minimum at x̄ and hence the derivative of f(x1,x2) in that direction must be zero. Below we define the derivative of a function in an arbitrary direction first and then repeat this claim. But before defining the directional derivative let me go over a definition that will play a pivotal role in our discussions:

Definition 5. The gradient of f(x1,x2, . . . ,xp) defined as

∇f(x1,x2, . . . ,xp) , [ ∂f

∂x1 (x̄),

∂f

∂x2 (x̄), . . . ,

∂f

∂xp (x̄)

]T .

Gradient plays a pivotal role in the analysis of maximas and minimas of a multivariable functions.

Definition 6. The directional derivative of a function f(x1,x2) in a direction (α1,α2) is defined as

D(α1,α2)f(x) , lim ∆t→0

f(x1 + α1∆t,x2 + α2∆t) −f(x1,x2) ∆t

.

Before we proceed it is important to gain some intuition for the directional derivative. First note that if (α1,α2) = (1, 0), then the derivative of the function in this direction is the same as

∂f ∂x1

. Can you explain

why? If (α1,α2) = (0, 1), then we get ∂f ∂x2

. However, by choosing (α1,α2) = (1, 1), we obtain the derivative of the function in a very different direction. Explain how the concept of directional derivative can be generalized for function f : Rp → R.

As described before we expect that for any (α1,α2), D(α1,α2)f(x̄) = 0. So, it seems that we can have infinitely many equations that x̄ should satisfy. It turns out that something interesting happens here. The following lemma explains this interesting fact:

3

Lemma 7. If all the partial derivatives of f : Rp → R are continuous, then1

Dαf(x) = ∇f(x)Tα.

Proof I’ll give you the proof sketch for p = 2. You will complete the proof in HW2 and also extend it to p > 2.

D(α1,α2)f(x) (a) = lim

∆t→0

f(x1 + α1∆t,x2 + α2∆t) −f(x1 + α1∆t,x2) ∆t

+ lim ∆t→0

f(x1 + α1∆t,x2) −f(x1,x2) ∆t

.

(2)

Explain in your HW why equality (a) is correct. Let’s calculate each limit. I start with the second limit. You should prove in your HW that

lim ∆t→0

f(x1 + α1∆t,x2) −f(x1,x2) ∆t

= α1 ∂f

∂x1 . (3)

The first limit in (2) is slightly more tricky. First prove that

f(x1 + α1∆t,x2 + α2∆t) −f(x1 + α1∆t,x2) = ∂f

∂x2 (x1 + α1∆t,z)α2∆t,

where z ∈ [x2,x2 + α2∆t]. Can you explain where this equation comes from? Now use this equation and prove that

lim ∆t→0

f(x1 + α1∆t,x2 + α2∆t) −f(x1 + α1∆t,x2) ∆t

= ∂f

∂x2 (x1,x2)α2. (4)

By combining (2), (3), and (4) we obtain what we wanted.

So according to this lemma once the gradient is zero, all the directional derivatives are zero. Why? Hence, we only need to check the gradient of the function, and once that is zero we can use the above theorem to say that in all different directions the derivative is zero. We can essentially summarize the above discussion in the following lemma:

Lemma 8. If x̄ = (x̄1, . . . , x̄p) is a local minima (maxima) of f(x), then ∇f(x̄) = 0. Also, if ∇f(x̄) = 0 the derivative of f(x̄) at x̄ in any other direction is equal to zero.

Proof Try to summarize the above discussion and write it as a proof for yourself here.

Example 2 Where is the minima of the function f(x1,x2) = (x1 − 1)2 + (x2 − 2)2 + x1x2? To answer this question we first check the gradient condition. At the minimum we know that the gradient of f(x1,x2) should be zero. Hence,

∂f(x1,x2)

∂x1 (x̄) = 2(x̄1 − 1) + x̄2 = 0.

∂f(x1,x2)

∂x2 (x̄) = 2(x̄2 − 1) + x̄1 = 0.

By solving these two linear equations we obtain x̄1 = x̄2 = 2/3.

1This theorem is true for more general conditions. Essentially as long as a function is differentiable this statement is true. But, we do not want to spend much time on the calculus details and hence we assume continuity of partial derivatives.

4

Remark Note that similar to the univariable case the fact that ∇f(x̄) = 0 does not guarantee that x̄ is a mininum of the function. It could be a minimum or maximum or even a saddle-point. In order to ensure that x̄ is a minimum we have to check the Hessian of the function, which can be considered intuitively as the second order derivative of multivariable functions. I encourage you to read about Hessian and its properties, but I do not review it here and we will not use it in the course.

Let me go over some important examples. These examples will be used many times in this course. I encourage you memorize these examples, as for instance you remember the derivative of f(x) = xn in the univariable case. You do not want to calculate them from scratch every time you see them.

Example 3 Let x ∈ Rp. What is the gradient of f(x) = c, where c is a fixed real number? Note that all the partial derivatives are zero. Hence, ∇f(x) = 0, where 0 is a p-dimensional vector whose entries are all equal to zero.

Example 4 Let c ∈ Rp denote a fixed vector. What is the gradient of f(x) = cT x? Note that f(x) =

∑ cixi. Hence, for any j ∈{1, 2, 3, . . . ,p} we have

∂f(x)

∂xj = cj.

Therefore, ∇f(x) = c.

Example 5 Consider the function f : Rp → R, f(x) = xTAx, where A denotes a fixed matrix. What is the gradient of this function? Note that

∂f

∂xk =

∂( ∑ i

∑ j Aijxixj)

∂xk

(a) =

∂( ∑ j Akjxkxj + ∂

∑ i 6=k Aikxkxi)

∂xk

(b) = 2Akkxk +

∑ j 6=k

Akjxj + ∑ i 6=k

Aikxi =

p∑ j=1

Akjxj +

p∑ i=1

Aikxi. (5)

Explain why Equality (a) holds. Explain why Equality (b) holds. Use the last equation and convince yourself that ∇f(x) = (A + AT )x. What happens when A is a symmetric matrix? Is it similar to a formula that you saw in univariate functions?

Example 6 Functions of the form f : Rp → R, f(x) = xTAx + bTx + c are called quadratic functions. Can you find the gradient of these functions?

5

x1

x2

f (x1, x2)

a b

c

Figure 2: Diagram of f(x1, x2).

6