design and analysis of algorithms -...

1

Design and Analysis of Algorithms

演算法設計與分析

Lecture 2March 4, 2010洪國寶

2

Homework # 1

1. 2.1-3 (p. 22)2. 2.2-3 (p. 29)3. 2.3-6 (p. 37)4. 2-2 (p. 40)

Due March 11, 2010

3

Outline

• Review• Introduction to design• Recurrences

4

Course information (5/5)

• Grading (Tentative)Homework 25%

(You may collaborate when solving the homework, however when writing up the solutions you must do so on your own. Handwritten only.)

Midterm exam 30% (Open book and notes)Final exam 30% (Open book and notes)Class participation 15%

5

Review: Definition of algorithms

• An algorithm is any well-defined computational procedure that takes some value, or set of values, as input and produces some value, or set of values, as output.

• The goal of this course is to explore some paradigms of designing efficient and correctalgorithms for computational problems.

6

Motivation (6/6)

• The importance of efficient algorithms– Total system performance depends on choosing

efficient algorithms as much as on choosing fast hardware.

– A faster algorithm running on a slower computer will always win for sufficiently large instances.

7

Review: Correct algorithms

• An algorithm is said to be correct if for every input instance, it halts with correct output.

• Correctness is not obvious.

8

Review: well-specified computational problem

• Example of a well-specified computational problem: sorting problem (p. 5)

Input: sequence ⟨a1, a2, …, an⟩ of numbers.Output: permutation ⟨a'1, a'2, …, a'n⟩ such

that a'1 ≤ a'2 ≤ … ≤ a'n .

9

Insertion sort

10

Review: Loop invariant

• A Method for Proving the Correctness of Iterative Algorithms (that is, algorithms contain loop structures)

• Similar to mathematical induction

Loop Invariants

• A loop invariant states a condition that is true immediately before every iteration of a loop.• The invariant should be true also immediately after the last iteration.

Time (# of computation steps)

It. 1 It. 2 It. 3 It. 4 · · · It. k

Loop Invariants• We have to show:–Initialization:

The loop invariant holds before the first iteration

–Maintenance:If the loop invariant holds before the i-th iteration, it holds

before the (i+1)-st iteration.

–Termination:If the loop invariant holds after the last iteration, the

algorithm does what we want.

A Loop Invariant for Insertion Sort

• Invariant: Before the iteration of the outer loop where j = k + 1, array A[1..k] stores the k elements initially in A[1..k] in sorted order. Array A[k+1..n]is unaltered.

14

Review: Algorithm analysis

• Analysis: predict the cost of an algorithm in terms of resources and performance

• Resource: time, space, bandwidth, random bits, etc.

15

Review: Algorithm analysis• Machine Model

Generic Random Access Machine (RAM)Executes operations sequentiallyEach memory access takes exactly 1 stepSet of primitive operations:

-- Arithmetic. Logical, Comparisons, Function calls

Loop and subroutine calls are not primitive ops.• Simplifying assumption: all ops cost 1 unit

16

Review: Algorithm analysis

• Under the RAM model, the running time of an algorithm depends mainly on input size.

• Input size: depends on the problem- number of items (sorting problem)- number of bits (number theoretic problems)- number of edges and vertices (graph problems)

17

Review: Algorithm analysis

• Reasons for worst case analysis- upper bound- worst case occurs fairly often- average case is often roughly as bad as worst case

18

Notations

f(n) >> c g(n)f(n) = ω(g(n))Little Omega

f(n) << c g(n)f(n) = o(g(n))Little Oh

f(n) ≥ c g(n)f(n) = Ω(g(n))Omega

f(n) ≤ c g(n)f(n) = O(g(n))BigOh

f(n) ≈ c g(n)f(n) = θ(g(n))Theta

19

Upper Bound Notation• We say InsertionSort’s run time is O(n2)

– Properly we should say run time is in O(n2)– Read O as “Big-O” (you’ll also hear it as

“order”)• In general a function

– f(n) is O(g(n)) if there exist positive constants cand n0 such that f(n) ≤ c ⋅ g(n) for all n ≥ n0

• Formally– O(g(n)) = f(n): ∃ positive constants c and n0

such that f(n) ≤ c ⋅ g(n) ∀ n ≥ n0

20

Asymptotic Tight Bound

• A function f(n) is Θ(g(n)) if ∃ positive constants c1, c2, and n0 such that

c1 g(n) ≤ f(n) ≤ c2 g(n) ∀ n ≥ n0

• Theorem– f(n) is Θ(g(n)) iff f(n) is both O(g(n)) and

Ω(g(n))

21

Introduction to Analysis

• Asymptotic notationsExamples: use blackboard

22

Introduction to Analysis• Asymptotic notation properties: (pp. 48-50)

For the following, assume that f(n) and g(n) are asymptotically positive. – Transitivity:

f (n) = Θ(g(n)) and g(n) = Θ(h(n)) imply f (n) = Θ(h(n)), f (n) = O(g(n)) and g(n) = O(h(n)) imply f (n) = O(h(n)), f (n) = Ω(g(n)) and g(n) = Ω(h(n)) imply f (n) = Ω(h(n)),

– Reflexivity: f (n) = Θ(f(n)), f(n) = O(f(n)), f(n) = Ω(f(n)), – Symmetry: f(n) = Θ(g(n)) if and only if g(n) = Θ(f(n)).

• Although any two real numbers can be compared, not all functions are asymptotically comparable.

– That is, for two functions (n) and g(n), it may be the case that neither f (n) = O(g(n)) nor f(n) = Ω(g(n)) holds.

Q: Is Θ an equivalence relation?How about O and Ω?

23

Outline

• Review• Introduction to design• Recurrences

24

Introduction to Design (1/)

• There are many ways to design algorithms- induction (incremental approach):insertion sort

- divide and conquer:merge sort

25

Introduction to Design (2/)

• divide and conquer:1. divide the problem into a number of

subproblems2. conquer the subprobelms (recursively)3. combine the solutions to the subproblems

into the solution of the original problem

26

Introduction to Design (3/)

• merge sort1. divide the sequence into two

subsequences of n/2 elements each2. sort the two subsequences recursively

using merge sort3. merge the two sorted subsequences

27

Introduction to Design (4/)

• merge sortTo sort the two subsequences recursivelyusing merge sort, we need to specify the indexes of the subsequences.

Merge-Sort (A, p, r)

28

Merge-Sort (A, p, r)INPUT: a sequence of n numbers stored in

array AOUTPUT: an ordered sequence of n numbers

1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)

divide

conquer

combine

29

Merge (A, p, q, r)INPUT: two sorted sequences A[p..q], A[q+1..r]OUTPUT: an ordered sequence A[p..r]

1. Create new arrays L and R2. Copy A[p..q] to L and copy A[q+1..r] to R3. Copy the elements in L and R back to A[p,r]

in order

30

.

31

The Merge Procedure: Repeatedly copy the smallest

element from L and R to A:10 i← 111 j← 112 for k = p to r13 do if L[i] ≤ R[j]14 then A[k] ← L[i]15 i← i + 116 else A[k] ← R[j]17 j← j + 1

Copy A[p..q] and A[q + 1..r] to temporary arrays L and R:

1 n1← q – p + r2 n2← r – q3 Create array L and R4 for i = 1 to n15 do L[i] ← A[p + i – 1]6 for j = 1 to n27 do R[j] ← A[q + j]

Create two artificial endmarkers that are never copied to

A:8 L[n1 + 1] ←∞9 R[n2 + 1] ←∞ ｝Why?

Merge ProcedureRunning Time

Lemma: Procedure Merge takes at most cn time, wheren = r – p + 1 and c is an appropriate constant.

Merge ProcedureCorrectness

Lemma:If arrays A[p..q] and A[q + 1..r] are sorted before invoking procedure Merge,

array A[p..r] is sorted after procedure Merge finishes.

Q: What is the most appropriate loop invariant for the merge procedure?

35Let’s take a closer look at Figure 2.3.

Merge ProcedureCorrectness

Lemma: If arrays A[p..q] and A[q + 1..r] are sorted beforeinvoking procedure Merge, array A[p..r] is sorted after procedure Merge finishes.

Loop invariant: Before every iteration of the last for-loop, array A[p..k – 1] stores the k – p smallest elements in L and R. Moreover, L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A

37

Correctness of the Merge procedure

• Use blackboard

38

Introduction to Design (17/)

• Merge sort(A,p,r)1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)• To sort a sequence of n numbers stored in

array A, run merge-sort(A,1,n).Example (Figure 2.4)

39

Merge sort(A,1,8)

Merge sort(A,1,4) Merge sort(A,5,8)

Merge sort(A,1,2)

p=r

40

Introduction to Design (19/)

• merge sort (remaining issues)- correctness- running time- compared to insertion sort

What are the advantages of merge sort?What are the advantages of insertion sort?

41

Correctness of merge sort

• By induction on length s of a segment of the array A wheres = r – p + 1

Merge-sort(A,p,r)1. if p < r2. then q ← [(p+r)/2]3. Merge-Sort (A, p, q)4. Merge-Sort (A, q+1, r)5. Merge (A, p, q, r)

42

Correctness of merge sort

• Use blackboard

43

Merge-SortTo sort a sequence of n numbers stored in array A, run merge-sort(A, 1, n).

if 1 < nthen q ← [(1+n)/2]

Merge-Sort (A, 1, q)Merge-Sort (A, q+1, n)Merge (A, 1, q, n)

How do we find the running time?

44

Analysis of Merge Sort• Divide: computing the middle takes Θ(1)• Conquer: solving 2 subproblems takes 2T(n/2)• Combine: merging n elements takes Θ(n)• Total:

T(n) = Θ(1) if n = 1T(n) = 2T(n/2) + Θ(n) if n > 1

⇒ T(n) = Θ(n lg n) (Chapter 4)

Sorting (Recap)

• We have studied two sorting algorithms:– Insertion Sort takes c1 n2 time.– Merge Sort takes c2 n lg n time.• Even if c2≫ c1, Merge Sort will ultimately outperform Insertion Sort– Example (use blackboard)

Sort n = 106 numbers, c1 = 2, and c2 = 50

46

Motivation (6/6)

• The importance of efficient algorithms– Total system performance depends on choosing

efficient algorithms as much as on choosing fast hardware.

– A faster algorithm running on a slower computer will always win for sufficiently large instances.

47

Sorting (Recap)

• Two different paradigms:– Insertion Sort is an incremental algorithm:

• Maintain a solution for part of the input• Augment/correct the solution as we take more and more input

elements into account.– Merge Sort uses the divide-and-conquer paradigm:

• Divide the problem into two (or more) subproblems of approximately equal size/complexity

• Solve (conquer) the subproblems recursively• Combine the solutions for the subproblems into a solution for

the whole problem

48

Outline

• Review• Introduction to analysis (cont.)• Introduction to design• Recurrences

49

Recurrence Relations• Describe functions in terms of their values on

smaller inputs• Solution Methods (Chapter 4)

– Substitution Method– Iteration Method– Master Method

• Arise from Divide and ConquerT(n) = Θ(1) if n ≤ cT(n) = a T(n/b) + D(n)+C(n) otherwise

50

Recurrence Relations• Remarks

- T(n) is defined when n is an integer (the size of the input is always an integer)

- Generally, T(n) = O(1) for sufficiently small n

- When we state and solve recurrences, we often omit floors, ceilings, and boundary conditions.

51

Recurrence Relations• Describe functions in terms of their values on

smaller inputs• Solution Methods (Chapter 4)

– Substitution Method– Iteration Method– Master Method

52

Substitution Method• Guess the form of the solution, then use

mathematical induction to show that it works

• Works well when the solution is easy to guess

• No general way to guess the correct solution

53

An ExampleTo Solve: T(n) = 3T(n/3) + n • Guess: T(n) = O(n lg n) ≤ c n lg n • Calculate:

T(n) ≤ 3c n/3 lg n/3 + n≤ c n lg (n/3) + n = c n lg n - c n lg3 + n= c n lg n - n (c lg 3 - 1)≤ c n lg n

(The last step is true for c ≥ 1 / lg3.)

Q: Is this mathematical induction?

54

Making a Good Guess• Guess a similar solution to one that you have seen

before– T(n) = 3T(n/3 + 5) + n and T(n) = 3T(n/3) + n

When n is large, the difference between n/3 and (n/3 + 5) is insignificant

• Another way is to prove loose upper and lower bounds on the recurrence and then reduce the range of uncertainty.– Start with T(n) = Ω(n) & T(n) = O(n2) ⇒ T(n) = Θ (n log n)

55

Subtleties• When the math doesn’t quite work out in the induction,

try to adjust your guess with a lower-order term. For example:

– We guess T(n) = O(n) ≤ c n for T(n) = 3T(n/3)+ 4, but we have T(n) ≤ 3c n/3 + 4 = c n + 4

56

Subtleties• When the math doesn’t quite work out in the induction,

try to adjust your guess with a lower-order term. For example:

– We guess T(n) = O(n) ≤ c n for T(n) = 3T(n/3)+ 4, but we have T(n) ≤ 3c n/3 + 4 = c n + 4

– New guess is T(n) ≤ c n - b, where b ≥ 0T(n) ≤ 3(c n/3 - b)+4 = c n - 3b + 4 = c n - b - (2b-4)Therefore, T(n) ≤ c n - b, if 2b - 4 ≥ 0 or if b ≥ 2

57

Changing Variables• Use algebraic manipulation to turn an unknown

recurrence into one similar to what you have seen before. – Consider T(n) = 2T(n1/2) + lg n – Rename m = lg n (that is, n=2m) and we have

T(2m) = 2T(2m/2) + m – Set S(m) = T(2m) and we have

S(m) = 2S(m/2) + m ⇒ S(m) = O(m lg m)– Changing back from S(m) to T(n), we have

T(n) = T(2m) = S(m) = O(m lg m) = O(lg n lg lg n)

58

Avoiding Pitfalls• Be careful not to misuse asymptotic notation. For

example:

– We can falsely prove T(n) = O(n) by guessing T(n) ≤ c nfor T(n) = 2T(n/2) + n

T(n) ≤ 2c n/2 + n≤ c n + n = O(n) ⇐ Wrong!

– The error is that we haven’t proved T(n) ≤ c n

59

Recurrence Relations• Describe functions in terms of their values on

smaller inputs• Solution Methods (Chapter 4)

– Substitution Method– Iteration Method– Master Method

60

Iteration Method• Expand (iterate) the recurrence and express it as a

summation of terms dependent only on n and the initial conditions

• The key is to focus on 2 parameters– the number of times the recurrence needs to be iterated to

reach the boundary condition– the sum of terms arising from each level of the iteration

process• Techniques for evaluating summations can then be used

to provide bounds on solution.

61

An Example• Solve: T(n) = 3T(n/4) + n

T(n) = n + 3T(n/4)= n + 3[ n/4 + 3T(n/16) ]= n + 3[n/4] + 9T(n/16)= n + 3[n/4] + 9 [n/16] + 27T(n/64)

T(n) ≤ n + 3n/4 + 9n/16 + 27n/64 + … + 3log4 nΘ(1)≤ n ∑ (3/4)i + Θ(nlog43)= 4n+ o(n)= O(n)

62

Another Example• Insertion sort (recursive version)

T(n) = T(n-1) + n• Solve: T(n) = T(n-1) + n

T(n) = T(n-1) + n = (T(n-2) + n-1)+ n = T(n-3) + n -2 + (n-1) + n = 1+2+3+ … + n

= n(n+1)/2= O(n2)

Insertion-sort(A,n)if n=1 then return A[n]

else Insertion-sort(A,n-1)Insert(A, n-1, A[n])

63

Recursion Trees

• Keep track of the time spent on the subproblems of a divide and conquer algorithm

• A convenient way to visualize what happens when a recursion is iterated

• Help organize the algebraic bookkeeping necessary to solve the recurrence

64

Merge Sortn

n/2

n/4 n/4

n/2

n/4 n/4

Running times to merge two sublists

Running time to sort the left sublist

65

Running Timen n=n

n/2

n/4 n/4

n/2

n/4 n/4

2x(n/2) = n

lg n4x(n/4) = n

Total: n lg n

66

Recursion Trees and Recurrences• Useful even when a specific algorithm is not specified

– For T(n) = 2T(n/2) + n2, we have

67

Recursion Trees

T(n) = Θ(n2)

68

Recursion Trees• For T(n) = T(n/3) + T(2n/3) + n

T(n) = O(n lg n)

69

Merge sort revisited• In merge sort, we can divide the sequence into

two subsequences of n/3 and 2n/3 elements each.• Then we will result in the following recurrence

T(n) = T(n/3) + T(2n/3) + n.• The previous slide shows that the asymptotic

running time remains the same.

70

Recurrence Relations• Describe functions in terms of their values on

smaller inputs• Solution Methods (Chapter 4)

– Substitution Method– Iteration Method– Master Method

71

Master Method• Provides a “cookbook” method for solving

recurrences of the formT(n) = a T(n/b) + f(n)

• Assumptions:– a ≥ 1 and b ≥ 1 are constants

– f(n) is an asymptotically positive function

– T(n) is defined for nonnegative integers– We interpret n/b to mean either n/b or n/b

72

The Master Theorem• With the recurrence T(n) = a T(n/b) + f(n) as in the

previous slide, T(n) can be bounded asymptotically as follows:

1. If f(n)=O(nlogba-ε) for some constant ε > 0, then T(n)= Θ(nlogba).

2. If f(n) = Θ(nlogba), then T(n) = Θ(nlogba lg n).3. If f(n) = Ω ( nlogba+ε ) for some constant ε > 0,

and if a f(n/b) ≤ c f(n) for some constant c < 1and all sufficiently large n, then T(n)= Θ(f(n)).

73

Simplified Master TheoremLet a ≥ 1 and b > 1 be constants and let T(n) be the

recurrenceT(n) = a T(n/b) + c nk

defined for n ≥ 0.1. If a > bk, then T(n) = Θ( nlogba ).2. If a = bk, then T(n) = Θ( nk lg n ).3. If a < bk, then T(n) = Θ( nk ).

74

Proof of the master theorem

75

Proof of the master theorem

• Use blackboard

76

The Master Theorem (Recap)

• Given: a divide and conquer algorithm– An algorithm that divides the problem of size n

into a subproblems, each of size n/b– Let the cost of each stage (i.e., the work to

divide the problem + combine solved subproblems) be described by the function f(n)

• Then, the Master Theorem gives us a cookbook for the algorithm’s running time.

77

The Master Theorem (Recap)

• if T(n) = aT(n/b) + f(n) then

( )

( )

( )

( )

( )

( )

<>

<Ω=

Θ=

=

Θ

Θ

Θ

=

+

−

10

largefor )()/( AND )(

)(

)(

)(

log)(

log

log

log

log

log

c

nncfbnafnnf

nnf

nOnf

nf

nn

n

nT

a

a

a

a

a

b

b

b

b

b

ε

ε

ε

78

Examples• T(n) = 16T(n/4) + n

– a = 16, b = 4, thus nlogba = nlog416 = Θ(n2)– f(n) = n = O(nlog416 - ε ) where ε = 1 ⇒ case 1.– Therefore, T(n) = Θ(nlogba ) = Θ(n2)

• T(n) = T(3n/7) + 1– a = 1, b=7/3, and nlogba = nlog 7/3 1 = n0 = 1– f(n) = 1 = Θ(nlogba) ⇒ case 2.– Therefore, T(n) = Θ(nlogba lg n) = Θ(lg n)

79

Examples (Cont.)• T(n) = 3T(n/4) + n lg n

– a = 3, b=4, thus nlogba = nlog43 = O(n0.793)– f(n) = n lg n = Ω(nlog43 + ε ) where ε ≈ 0.2 ⇒ case 3.– Therefore, T(n) = Θ(f(n)) = Θ(n lg n)

• T(n) = 2T(n/2) + n lg n– a = 2, b=2, f(n) = n lg n, and nlogba = nlog22 = n– f(n) is asymptotically larger than nlogba, but not

polynomially larger. The ratio lg n is asymptotically less than nε for any positive ε. Thus, the Master Theoremdoesn’t apply here.

80

Homework # 1

1. 2.1-3 (p. 22)2. 2.2-3 (p. 29)3. 2.3-6 (p. 37)4. 2-2 (p. 40)

Due March 11, 2010