Algorithm design problem set
LE/EECS3101
Design and Analysis of Algorithms
Running Time Analysis
Karim Jahed
Algorithm Analysis
• We are looking to answer the question: ”How much resources does an algorithm requires?”
• A resource could be anything (e.g., storage), but we are mostly interested in the running time.
• We will assume the RAM model for all our analyses. • Instructions run sequentially. • All operations require constant time.
1
Runtime Analysis
• Assume each operation cost some constant c.
• Count how many times each operation is executed.
2
Runtime Analysis - FindMax
1 f u n f i n d M a x ( A [ 1 . . n ] ) :
2 max = A [ 1 ]
3 f o r i = 2 t o n
4 i f max < A [ i ] t h e n
5 max = A [ i ]
6 end
7 end
8 r e t u r n max
Line Cost Times
2 c1 1
3 c2 n
4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1
3
Runtime Analysis - FindMax
1 f u n f i n d M a x ( A [ 1 . . n ] ) :
2 max = A [ 1 ]
3 f o r i = 2 t o n
4 i f max < A [ i ] t h e n
5 max = A [ i ]
6 end
7 end
8 r e t u r n max
Line Cost Times
2 c1 1
3 c2 n
4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1
• Best case: k = 0, i.e., the maximum element is at A[1]
• Worst case: k = n − 1, i.e., max changes in each iteration (e.g., the array is sorted)
• Average case: ?
3
Best, Worst, and Average Case - FindMax
4
Best, Worst, and Average Case
• The running time of an algorithm typically grows with the input size.
• What is ‘input size’ ?
• Case analysis: • Best case: Not very informative. • Average case: Very useful but hard to determine. • Worst case: Easier to analyze and often crucial for applications in
certain domain (e.g., robotics, finance, etc).
5
Analysis of FindMax
1 f u n f i n d M a x ( A [ 1 . . n ] ) :
2 max = A [ 1 ]
3 f o r i = 2 t o n
4 i f max < A [ i ] t h e n
5 max = A [ i ]
6 end
7 end
8 r e t u r n max
Line Cost Times
2 c1 1
3 c2 n
4 c3 n − 1 5 c4 0 ≤ k ≤ n − 1 8 c5 1
• Best case: c1 + c5 −c3 + (c2 + c3)n • Worst case: c1 + c5 − c3 − c4 + (c2 + c3 + c4)n
6
Simplified Running Time Analysis
The worst-case running time for: c1 + c5 −c3 − c4 + (c2 + c3 + c4)n is
• Complex
• Not very useful as the ci ’s are machine dependent
• A simpler expression: C + Dn
• We can simply say this is linear time, i.e., n. Why?
7
Simplified Running Time Analysis - Rationale
• Discarding lower order terms & coefficients • Coefficients are machine dependent. • Cleaner and usually realistic.
• Caveat: We will never get: • Exact run times • Useful info for small inputs • Small differences in performance
8
Analysis of FindMax - Summary
• FindMax worst-case running time is C + Dn, written as Θ(n).
• We say FindMax runs in linear time.
• Can we do better?
• No. We will later show that for any algorithm for this problem, and for each n > 0, there exist an input that makes the algorithm take
Ω(n) time.
9
Analysis of FindMax - Summary
• FindMax worst-case running time is C + Dn, written as Θ(n).
• We say FindMax runs in linear time.
• Can we do better?
• No. We will later show that for any algorithm for this problem, and for each n > 0, there exist an input that makes the algorithm take
Ω(n) time.
9
Analysis of InsertionSort
1 f u n i n s e r t i o n S o r t (A [ 1 . . n ] )
2 f o r j = 2 t o n
3 k e y = A [ j ]
4 i = j − 1 5 w h i l e i > 0
6 and A [ i ] > k e y
7 A [ i + 1 ] = A [ i ]
8 i = i − 1 9 A [ i + 1 ] = k e y
Line Cost Times
2 c1 n
3 c2 n − 1 4 c3 n − 1 5 c4
∑n j=2 tj
7 c5 ∑n
j=2(tj − 1) 8 c6
∑n j=2(tj − 1)
9 c7 n − 1
• Best case is when the array is sorted: (c1 + c2 + c3 + c4 + c7)n − c2 −c3 − c7
10
Analysis of InsertionSort
1 f u n i n s e r t i o n S o r t (A [ 1 . . n ] )
2 f o r j = 2 t o n
3 k e y = A [ j ]
4 i = j − 1 5 w h i l e i > 0
6 and A [ i ] > k e y
7 A [ i + 1 ] = A [ i ]
8 i = i − 1 9 A [ i + 1 ] = k e y
Line Cost Times
2 c1 n
3 c2 n − 1 4 c3 n − 1 5 c4
∑n j=2 tj
7 c5 ∑n
j=2(tj − 1) 8 c6
∑n j=2(tj − 1)
9 c7 n − 1
• Best case is when the array is sorted: (c1 + c2 + c3 + c4 + c7)n − c2 −c3 − c7
10
Worst Case Analysis of InsertionSort
The worst case is when the array is sorted in reverse order. Each
element A[j] must be compared against the entire sorted sub-array
A[1..j-1]. Therefore, tj = j. Note that:
n∑ j=2
j = n(n + 1)
2 − 1
n∑ j=2
(j − 1) = n(n − 1)
2
Note: Refer to Appendix A of your text book for a refresher on how to
solve summations.
11
Worst Case Analysis of InsertionSort
The worst case is when the array is sorted in reverse order. Each
element A[j] must be compared against the entire sorted sub-array
A[1..j-1]. Therefore, tj = j. Note that:
n∑ j=2
j = n(n + 1)
2 − 1
n∑ j=2
(j − 1) = n(n − 1)
2
Note: Refer to Appendix A of your text book for a refresher on how to
solve summations.
11
Worst Case Analysis of InsertionSort (2)
The worst case for InsertionSort is thus:
T (n) = c1n + c2(n − 1) + c3(n − 1) + c4( n(n + 1)
2 − 1)
+ c5( n(n − 1)
2 ) + c6(
n(n − 1) 2
) + c7(n − 1)
• T (n) can be expressed as an2 + bn + c for some constants a, b, and c.
• Insertion sort thus requires quadratic time, i.e., T (n) ∈ Θ(n2)
12
Worst Case Analysis of InsertionSort (2)
The worst case for InsertionSort is thus:
T (n) = c1n + c2(n − 1) + c3(n − 1) + c4( n(n + 1)
2 − 1)
+ c5( n(n − 1)
2 ) + c6(
n(n − 1) 2
) + c7(n − 1)
• T (n) can be expressed as an2 + bn + c for some constants a, b, and c.
• Insertion sort thus requires quadratic time, i.e., T (n) ∈ Θ(n2)
12
Another example - Prefix Average
The ith prefix average of an array A is the average of the first i + 1
elements of A, i.e.,
S[i] = (A[1] + A[2] ... A[i + 1]) / (i + 1)
13
Prefix Average - A slow algorithm
1 f u n s l o w P r e f i x A v e r a g e ( A [ 0 . . n − 1 ] ) : 2 f o r j = 0 t o n − 1 : 3 t o t a l = 0
4 f o r i = 0 t o j :
5 t o t a l = t o t a l + A [ i ]
6 end
7 S [ j ] = t o t a l / ( j + 1 )
8 end
14
Prefix Average - A slow algorithm
1 f u n s l o w P r e f i x A v e r a g e ( A [ 0 . . n − 1 ] ) : 2 f o r j = 0 t o n − 1 : 3 t o t a l = 0
4 f o r i = 0 t o j :
5 t o t a l = t o t a l + A [ i ]
6 end
7 S [ j ] = t o t a l / ( j + 1 )
8 end
• The inner loop body takes Θ(1) steps.
• The inner loop is executed j + 1 times.
• The outer loop is executed n times.
• Let’s sum it all up.
14
Prefix Average - Running time
T (n) = n−1∑ j=0
j∑ i=0
1
= n−1∑ j=0
(j + 1)
= n∑
j=1
j
= n(n + 1)/2
Therefore, T (n) ∈ Θ(n2)
15
Prefix Average - A linear time algorithm
1 f u n f a s t P r e f i x A v e r a g e (A [ 0 . . n − 1 ] ) : 2 t o t a l = 0
3 f o r j = 0 t o n − 1 : 4 t o t a l = t o t a l + A [ j ]
5 S [ j ] = t o t a l / ( j + 1 )
6 end
16