Algorithm design problem set

LE/EECS3101

Design and Analysis of Algorithms

Running Time Analysis

Karim Jahed

Algorithm Analysis

• We are looking to answer the question: ”How much resources does an algorithm requires?”

• A resource could be anything (e.g., storage), but we are mostly interested in the running time.

• We will assume the RAM model for all our analyses. • Instructions run sequentially. • All operations require constant time.

1

Runtime Analysis

• Assume each operation cost some constant c.

• Count how many times each operation is executed.

2

Runtime Analysis - FindMax

1 f u n f i n d M a x ( A [ 1 . . n ] ) :

2 max = A [ 1 ]

3 f o r i = 2 t o n

4 i f max < A [ i ] t h e n

5 max = A [ i ]

6 end

7 end

8 r e t u r n max

Line Cost Times

2 c1 1

3 c2 n

4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1

3

Runtime Analysis - FindMax

1 f u n f i n d M a x ( A [ 1 . . n ] ) :

2 max = A [ 1 ]

3 f o r i = 2 t o n

4 i f max < A [ i ] t h e n

5 max = A [ i ]

6 end

7 end

8 r e t u r n max

Line Cost Times

2 c1 1

3 c2 n

4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1

• Best case: k = 0, i.e., the maximum element is at A[1]

• Worst case: k = n − 1, i.e., max changes in each iteration (e.g., the array is sorted)

• Average case: ?

3

Best, Worst, and Average Case - FindMax

4

Best, Worst, and Average Case

• The running time of an algorithm typically grows with the input size.

• What is ‘input size’ ?

• Case analysis: • Best case: Not very informative. • Average case: Very useful but hard to determine. • Worst case: Easier to analyze and often crucial for applications in

certain domain (e.g., robotics, finance, etc).

5

Analysis of FindMax

1 f u n f i n d M a x ( A [ 1 . . n ] ) :

2 max = A [ 1 ]

3 f o r i = 2 t o n

4 i f max < A [ i ] t h e n

5 max = A [ i ]

6 end

7 end

8 r e t u r n max

Line Cost Times

2 c1 1

3 c2 n

4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1

• Best case: c1 + c5 −c3 + (c2 + c3)n • Worst case: c1 + c5 − c3 − c4 + (c2 + c3 + c4)n

6

Simplified Running Time Analysis

The worst-case running time for: c1 + c5 −c3 − c4 + (c2 + c3 + c4)n is

• Complex

• Not very useful as the ci ’s are machine dependent

• A simpler expression: C + Dn

• We can simply say this is linear time, i.e., n. Why?

7

Simplified Running Time Analysis - Rationale

• Discarding lower order terms & coefficients • Coefficients are machine dependent. • Cleaner and usually realistic.

• Caveat: We will never get: • Exact run times • Useful info for small inputs • Small differences in performance

8

Analysis of FindMax - Summary

• FindMax worst-case running time is C + Dn, written as Θ(n).

• We say FindMax runs in linear time.

• Can we do better?

• No. We will later show that for any algorithm for this problem, and for each n > 0, there exist an input that makes the algorithm take

Ω(n) time.

9

Analysis of FindMax - Summary

• FindMax worst-case running time is C + Dn, written as Θ(n).

• We say FindMax runs in linear time.

• Can we do better?

• No. We will later show that for any algorithm for this problem, and for each n > 0, there exist an input that makes the algorithm take

Ω(n) time.

9

Analysis of InsertionSort

1 f u n i n s e r t i o n S o r t (A [ 1 . . n ] )

2 f o r j = 2 t o n

3 k e y = A [ j ]

4 i = j − 1 5 w h i l e i > 0

6 and A [ i ] > k e y

7 A [ i + 1 ] = A [ i ]

8 i = i − 1 9 A [ i + 1 ] = k e y

Line Cost Times

2 c1 n

3 c2 n − 1 4 c3 n − 1 5 c4

∑n j=2 tj

7 c5 ∑n

j=2(tj − 1) 8 c6

∑n j=2(tj − 1)

9 c7 n − 1

• Best case is when the array is sorted: (c1 + c2 + c3 + c4 + c7)n − c2 −c3 − c7

10

Analysis of InsertionSort

1 f u n i n s e r t i o n S o r t (A [ 1 . . n ] )

2 f o r j = 2 t o n

3 k e y = A [ j ]

4 i = j − 1 5 w h i l e i > 0

6 and A [ i ] > k e y

7 A [ i + 1 ] = A [ i ]

8 i = i − 1 9 A [ i + 1 ] = k e y

Line Cost Times

2 c1 n

3 c2 n − 1 4 c3 n − 1 5 c4

∑n j=2 tj

7 c5 ∑n

j=2(tj − 1) 8 c6

∑n j=2(tj − 1)

9 c7 n − 1

• Best case is when the array is sorted: (c1 + c2 + c3 + c4 + c7)n − c2 −c3 − c7

10

Worst Case Analysis of InsertionSort

The worst case is when the array is sorted in reverse order. Each

element A[j] must be compared against the entire sorted sub-array

A[1..j-1]. Therefore, tj = j. Note that:

n∑ j=2

j = n(n + 1)

2 − 1

n∑ j=2

(j − 1) = n(n − 1)

2

Note: Refer to Appendix A of your text book for a refresher on how to

solve summations.

11

Worst Case Analysis of InsertionSort

The worst case is when the array is sorted in reverse order. Each

element A[j] must be compared against the entire sorted sub-array

A[1..j-1]. Therefore, tj = j. Note that:

n∑ j=2

j = n(n + 1)

2 − 1

n∑ j=2

(j − 1) = n(n − 1)

2

Note: Refer to Appendix A of your text book for a refresher on how to

solve summations.

11

Worst Case Analysis of InsertionSort (2)

The worst case for InsertionSort is thus:

T (n) = c1n + c2(n − 1) + c3(n − 1) + c4( n(n + 1)

2 − 1)

+ c5( n(n − 1)

2 ) + c6(

n(n − 1) 2

) + c7(n − 1)

• T (n) can be expressed as an2 + bn + c for some constants a, b, and c.

• Insertion sort thus requires quadratic time, i.e., T (n) ∈ Θ(n2)

12

Worst Case Analysis of InsertionSort (2)

The worst case for InsertionSort is thus:

T (n) = c1n + c2(n − 1) + c3(n − 1) + c4( n(n + 1)

2 − 1)

+ c5( n(n − 1)

2 ) + c6(

n(n − 1) 2

) + c7(n − 1)

• T (n) can be expressed as an2 + bn + c for some constants a, b, and c.

• Insertion sort thus requires quadratic time, i.e., T (n) ∈ Θ(n2)

12

Another example - Prefix Average

The ith prefix average of an array A is the average of the first i + 1

elements of A, i.e.,

S[i] = (A[1] + A[2] ... A[i + 1]) / (i + 1)

13

Prefix Average - A slow algorithm

1 f u n s l o w P r e f i x A v e r a g e ( A [ 0 . . n − 1 ] ) : 2 f o r j = 0 t o n − 1 : 3 t o t a l = 0

4 f o r i = 0 t o j :

5 t o t a l = t o t a l + A [ i ]

6 end

7 S [ j ] = t o t a l / ( j + 1 )

8 end

14

Prefix Average - A slow algorithm

1 f u n s l o w P r e f i x A v e r a g e ( A [ 0 . . n − 1 ] ) : 2 f o r j = 0 t o n − 1 : 3 t o t a l = 0

4 f o r i = 0 t o j :

5 t o t a l = t o t a l + A [ i ]

6 end

7 S [ j ] = t o t a l / ( j + 1 )

8 end

• The inner loop body takes Θ(1) steps.

• The inner loop is executed j + 1 times.

• The outer loop is executed n times.

• Let’s sum it all up.

14

Prefix Average - Running time

T (n) = n−1∑ j=0

j∑ i=0

1

= n−1∑ j=0

(j + 1)

= n∑

j=1

j

= n(n + 1)/2

Therefore, T (n) ∈ Θ(n2)

15

Prefix Average - A linear time algorithm

1 f u n f a s t P r e f i x A v e r a g e (A [ 0 . . n − 1 ] ) : 2 t o t a l = 0

3 f o r j = 0 t o n − 1 : 4 t o t a l = t o t a l + A [ j ]

5 S [ j ] = t o t a l / ( j + 1 )

6 end

16