Algorithm design problem set

LE/EECS3101: Design and Analysis of Algorithms

Weekly Digest

Weeks 2-3: Runtime Analysis & Divide and

Conquer

1 Case Analysis vs Asymptotic Bounds

There is often confusion between ’case analysis’ (best/worst case) and ’asymp- totic bounds’ (Big-O, Omega, Theta). Big-O doesn’t necessarily mean worst case. We often use Big-O as an upper bound on the worse case, but we could also use it as an upper bound on the best case. Big-O and Omega are just functions (say, g(n)) that bind the running time (say, f(n)). O(g(n)) puts a bound on f(n) from above. As the input size approaches infinity, f(n) might get closer and closer to g(n) but never crosses it. Ω(g(n)) puts a bound on f(n) from below. f(n)can never be less than g(n). Now f(n), the running time, can be that of the worst case, best case, or even average case. Often, we are interested in placing an upper bound on the worst case, which is why Big-O is often associated with the worst case.

E.g., in Insertion Sort, the running time of the worst case is in O(n2) (bounded from above by g(n) = n2). In the worst case, the running time is also in Ω(n2) (bounded from below by g(n) = n2). Since the running time of the worst case is bounded from above and below by g(n) = n2, we say that we have a tight bound, i.e., in the worst case, Insertion Sort is in Θ(n2).

Similarly, the running time of Insertion Sort in the best case (array is sorted) is in (n) (also in (n2), (n3), (2n)...). The running time of the best case is also in Ω(n). Therefore, the running time of Insertion Sort in the best case is in Θ(n).

Again, we are often interested in the worst case. So when we are talking about the complexity or running time of an algorithm we are referring to the worst-case unless otherwise stated.

1

2 Runtime Analysis

We can compute the ’exact’ running time of an algorithm by summing the total cost of each line of code. The cost of a line of code is the number of times that line is executed multiplied by some constant cost c. The result of the summation is a polynomial.

Often, we are interested in the asymptotic running time, that is the running time as the input grows larger and larger. As the input size approaches infinity, the contribution of the lower-order terms and constants becomes minuscule, and the majority of the running time is taken by the highest- order term. Therefore, we often ignore the constants and lower-order terms and take the highest-order term as an upper bound.

We can compute the running time and derive an asymptotic upper, lower, or tight bound for the best, worst, and average cases. Although the average case is the most informative, it is pretty hard to define. An upper bound on the worst-case guarantees that the running time will never be polynomially larger (i.e., by a polynomial factor) than the bound.

3 Recurrences

Runtime analysis becomes more complicated when we add recursion into the mix. The cost of a recursive call is unknown. It is, in fact, the thing we are trying to compute! We can, however, write the running time in terms of itself. We call such expressions recurrence relations. Once we determine a recurrence relation, there are three main ways we can try to solve it:

• Guess and check: If we have a good guess, we can prove our guess is correct by induction over the size of the input n. Let T(n) = f(n) be our guess. The inductive steps assumes T (m) = f(m) is true for all m < n, and proves T (n) is true. We can guess the running time by examining the recursion tree or by forward substitution (trying out the first few terms, i.e., n = 1, n = 2, etc.)

• Backward substitution: Keep on substituting T (n) until a pattern is detected. Write the pattern in terms of the number of substitutions i. Compute the value of i, k, for which T (n) = T (1). Finally, substitute T (n) for T (1) and i for k.

• The Master Method: The Master method if useful in solving recur- rences of the form T (n) = aT (n/b) + f(n). This is often the case for divide and conquer algorithms where a is the number of subproblems, b is the factor by which the size of each subproblem is reduced, and f(n) is the time to divide and combine.

2

The recursive tree for a recurrence of the form T (n) = aT (n/b) + f(n) has logb n levels and a branching factor of a, i.e., it has a

i nodes at level i. The total number of base cases in such tree is nlogb a. The Master theorem defines three cases based on how much the time to solve the base cases takes up from the total computation time:

– Case 1: Time to solve the base cases is polynomially larger than the time to divide and combine, in other words, the time to divide and combine f(n) is bounded from above by the time to solve the bases cases. In this case T (n) = Θ(nlogb a).

– Case 2: Time to solve the base cases is roughly the same as the time to divide and combine, in other words, the time to divide and combine f(n) is bounded from above and below by the time to solve the bases cases. In this case T (n) = Θ(nlogb a log n).

– Case 3: Time to solve the base cases is polynomially smaller than the time to divide and combine, in other words, the time to divide and combine f(n) is bounded from below by the time to solve the bases cases. In this case T (n) = Θ(f(n)).

4 Divide and Conquer

A divide and conquer algorithm solves a large problem by recursively divid- ing it into smaller, more manageable subproblems, and then combining the solutions of the smaller subproblems to construct a solution to the larger problem. There are four questions to ask when designing a divide and con- quer algorithm:

1. What is the input size? e.g., in a sorting algorithm, the input size is the number of elements in the array. For an algorithm that computes the product of two integers, the input size is the maximum number of bits of both integers.

2. What is the smallest problem instance that we know how to trivially solve? For sorting, the smallest problem is when n = 1, i.e., the array has one element. In that case, the array is trivially sorted.

3. How to recursively divide the problem to get to the smallest problem?

4. How to combine the results of the smaller subproblems and compute the result of the larger problem? Often questions 3 and 4 are inter- twined.

3