design and analysis of algorithms -...

1

Design and Analysis of

Algorithms

演算法設計與分析

Lecture 8

November 16, 2016

洪國寶

2

Outline

• Review

– Amortized analysis

• Advanced data structures

– Binary heaps

– Binomial heaps

– Fibonacci heaps

– Data structures for disjoint sets

3

Huffman code (Implementation)

• Time complexity and data structure:

Let S be the set of n weights (nodes).

Constructing a Huffamn code based on greedy strategy can be described as following

Repeat until |S|=1

Find two min nodes x and y from S and remove them from S

Construct a new node z with weight w(z)=w(x)+w(y) and

insert z into S

4

Huffman code (Implementation)

Why data structures are important?

• An algorithm for constructing Huffman code (Tree):

Repeat until |S|=1– Find two min nodes x and y from S and remove them from S

– Construct a new node z with weight w(z)=w(x)+w(y) and insert z into S

• The time complexity of the algorithm depends on how S is implemented.

• Data structure for S each iteration total

linked list O(n) O(1) O(n2)

Sorted array O(1) O(n) O(n2)

? O(log n) O(log n) O(n log n)

Priority Queues

• A priority queue is an abstract data type which is like a regular queue or stack data structure, but where additionally each element has a "priority" associated with it.

• In a priority queue, an element with high priority is served before an element with low priority.

• Priority queues are often implemented with heaps.

• Applications.– Dijkstra's shortest path algorithm.

– Prim's MST algorithm.

– Event-driven simulation.

– Huffman encoding. (Lecture 6)

– Heapsort.

– . . .

5

6

Mergeable Heaps

Support the following operations. • MAKE-HEAP() creates and returns a new heap containing no elements.

• INSERT(H, x) inserts node x, whose key field has already been filled in, into

heap H.

• MINIMUM(H) returns a pointer to the node in heap H whose key is minimum.

• EXTRACT-MIN(H) deletes the node from heap H whose key is minimum,

returning a pointer to the node.

• UNION(Hl, H2) creates and returns a new heap that contains all the nodes of

heaps H1 and H2. Heaps H1 and H2 are "destroyed" by this operation.

• DECREASE-KEY(H, x, k) assigns to node x within heap H the new key value k,

which is assumed to be no greater than its current key value.

• DELETE(H, x) deletes node x from heap H.

7

Dijkstra/Prim

1 make-heap

|V| insert

|V| extract-min

|E| decrease-key

Priority Queues

make-heap

Operation

insert

minimum

extract-min

union

decrease-key

delete

1

Binary

log N

1

log N

N

log N

log N

1

Binomial

log N

log N

log N

log N

log N

log N

1

Fibonacci *

1

1

log N

1

1

log N

1

Linked List

1

N

N

1

1

N

Mergeable Heaps

O(|E| + |V| log |V|)O(|E| log |V|)O(|V|2)

8

Binary Heap: Definition• Binary heap.

– Almost complete binary tree.

• filled on all levels, except last, where filled from left to right

– Min-heap ordered.

• every child greater than (or equal to) parent

06

14

78 18

81 7791

45

5347

6484 9983

9

Binary Heap: Properties

•Properties.

– Min element is in root.

– Heap with N elements has height = log2 N.

06

14

78 18

81 7791

45

5347

6484 9983

N = 14

Height = 3

10

Binary Heaps: Array Implementation

•Implementing binary heaps.– Use an array: no need for explicit parent or child pointers.

•Parent(i) = i/2

•Left(i) = 2i

•Right(i) = 2i + 1

06

14

78 18

81 7791

45

5347

6484 9983

1

2 3

4 5 6 7

8 9 10 11 12 13 14

11

Binary heap operations

• Make-heap

make-heap( )

allocate an array H on size N

heap-size[H] 0

• Minimum

minimum(H)

return H[1]

12

Inserting into a binary heap

• There are two constraints that must be met

– an almost complete tree

– the heap-order property

• We increase the size of the heap to make a

hole in the next spot

• It the item can be legally inserted in the hole,

then do so, otherwise move the parent value

down and try again

13

Binary Heap: Insertion example

•Insert element x (key = 42) into heap.

– Insert into next available slot.

– Bubble up until it's heap ordered.

06

14

78 18

81 7791

45

5347

6484 9983 42 next free slot

14





06

14

78 18

81 7791

45

5347

6484 9983 4242

swap with parent

15





06

14

78 18

81 7791

45

4247

6484 9983 4253

swap with parent

16





06

14

78 18

81 7791

42

4547

6484 9983 53

stop: heap ordered

17

Binary Heap: Insertion algorithm

• insert(H,k)

heap-size[H] heap-size[H]+1

i heap-size[H]

while i > 1 and H[parent(i)] > k

do H[i] H[parent(i)]

i parent(i)

H[i] k

• Time complexity = O(tree height) = O(log n)

18

Binary Heap: Decrease Key

•Decrease key of element x to k.


06

14

78 18

81 7791

42

4547

6484 9983 53

19

Binary Heap: Decrease Key

• decrease-key(H,i,k)

while i > 1 and H[parent(i)] > k

do H[i] H[parent(i)]

i parent(i)

H[i] k

• Time complexity = O(tree height) = O(log n)

20

Binary Heap: Extract-Min

• Delete minimum element from heap.

– Exchange root with rightmost leaf.

– Bubble root down until it's heap ordered.

• power struggle principle: better subordinate is promoted

06

14

78 18

81 7791

42

4547

6484 9983 53

21

Binary Heap: Extract-Min• Delete minimum element from heap.




53

14

78 18

81 7791

42

4547

6484 9983 06

22

Binary Heap: Extract-Min• Delete minimum element from heap.




53

14

78 18

81 7791

42

4547

6484 9983

exchange with left child

23


• Delete minimum element from heap.




14

53

78 18

81 7791

42

4547

6484 9983

exchange with right child

24


• Delete minimum element from heap.– Exchange root with rightmost leaf.

– Bubble root down until it's heap ordered.• power struggle principle: better subordinate is promoted

– Time complexity = O(tree height) = O(log n)

14

18

78 53

81 7791

42

4547

6484 9983

stop: heap ordered

25


• extract-min(H)

if heap-size[H] < 1

then error “heap underflow”

min H[1]

H[1] H[heap-size(H)]

heap-size[H] heap-size[H] -1

heapify(H, 1)

return min

Bubble root down until it's heap ordered.

26

Binary Heap: Heapify

• When heapify(H,i) is called, it is assumed that the

binary trees rooted at left(i) and right(i) are heaps,

but that A[i] may be larger than its children, thus

violating the heap property.

• The function of heapify is to let the value at A[i]

float down in the heap so that the subtree rooted at

index i becomes a heap.

• The running time is O(h) if the height of node i

is h.

27

Binary Heap: Heapify

• heapify(H,i)

l left(i)

r right(i)

if l ≦ heap-size[H] and H[l] < H[i]

then smallest l

else smallest i

if r ≦ heap-size[H] and H[r] < H[smallest]

then smallest r

if smallest ≠ i

then H[i] H[smallest]

heapify(H,smallest)

28

Binary Heap: Delete

• delete(H,i)

H[i] H[heap-size[H]]

heap-size[H] heap-size[H] – 1

heapify(H,i)

29

H1 H2

14

78 18

81 7791

11

6253

6484 9941

Binary Heap: Union• Union.

– Combine two binary heaps H1 and H2 into a single heap.

– No easy solution.

• (N) operations apparently required

– Can support fast union with fancier heaps.

30

Binary Heap: Union

• Union(H1,H2)

H make-heap()

heap-size[H] heap-size[H1] + heapsize[H2]

for i 1 to heap-size[H1]

do H[i] H1[i]

for i 1 to heap-size[H2]

do H[heap-size[H1]+i] H2[i]

for i heap-size[H]/2 down to 1

do heapify(H,i)

31

Binary Heap: Union

14

78

41 91

62 8453

18

7781

99 6411

14

78 18

81 779141

11

6253

6484 99

H1

H2

32

Binary Heap: Union

• Proof of correctness by loop invariant

– At the start of each iteration, each node i+1, i+2, …,n is the root of a heap

• Time complexity

– Simple upper bound:

• each call to HEAPIFY: O(lg n)

• O(n) such calls

• running time at most O(n lg n)

– A tighter bound is O(n).

33

Binary Heap: Union

• Time complexity

– A tighter bound is O(n). (Use blackboard)

34

Binary Heap: Heapsort

• To sort N items:

– Insert N items into a binary max-heap.

– Perform N extract-max operations.

• O(N log N) sort.

• No extra storage (in-place sort).

• Which one is better? Merge sort or heap sort?

35

Build-Max-Heap Operation

• We start by recognizing that if each subtree

of a node is a heap, then when we connect

the two subtrees, we can obtain a new heap

by finding where the root value should go

• All leaf nodes are, by definition, heaps

38

Example: The action of max-heapify(A,2)

40

Example: The operation of build-max-heap

41

Running Time of

BUILD-MAX-HEAP

• Simple upper bound:

– each call to MAX-HEAPIFY: O(lg n)

– O(n) such calls

– running time at most O(n lg n)

• A tighter bound is O(n).

44

Running Time of HEAPSORT

• Upper bound:

– Build max-heap: O(n)

– each call to MAX-HEAPIFY: O(log n)

• O(n) such calls

– running time O(n log n)

45

Some sorting algorithms

Insertion

sort

Merge

sort

Heapsort Quicksort

complexity O(n2) O(n log n) O(n log n) O(n log n)

In-place

sort

yes no yes

Data

structure

Linked

list

Linked

list

Array

Remark On-line

sorting

46

Sorting algorithms

• The ideal sorting algorithm would have the following properties: – Stable: Equal keys aren't reordered.

– Operates in place, requiring O(1) extra space.

– Worst-case O(n·lg(n)) key comparisons.

– Worst-case O(n) swaps.

– Adaptive: Speeds up to O(n) when data is nearly sorted or when there are few unique keys.

• There is no algorithm that has all of these properties, and so the choice of the optimal sorting algorithm depends on the application.

47

Animated Sorting Algorithms

• http://www.sorting-algorithms.com/

http://www.sorting-algorithms.com/

48

Priority Queues

make-heap

Operation

insert

minimum

extract-min

union

decrease-key

delete

1

Binary

log N

1

log N

N

log N

log N

1

Binomial

log N

log N

log N

log N

log N

log N

1

Fibonacci *

1

1

log N

1

1

log N

1

Linked List

1

N

N

1

1

N

Mergeable Heaps

Coming up next

49

Binomial Trees

• Binomial tree.

– Recursive definition:

Bk-1

Bk-1

B0 Bk

B0 B1 B2 B3 B4

50

Binomial Trees• Useful properties of order k binomial tree Bk.– Number of nodes = 2k.

– Height = k.

– Degree of root = k.

– Deleting root yields binomialtrees Bk-1, … , B0.

• Proof.– By induction on k.

B0 B1 B2 B3 B4

B1

Bk

Bk+1

B2

B0

51

Binomial Trees

• A property useful for naming the data structure.

– Bk has nodes at depth i.

B4

i

k

62

4

depth 2

depth 3

depth 4

depth 0

depth 1

52

Binomial Heap

• Binomial heap. Vuillemin, 1978.

– Sequence of binomial trees that satisfy binomial heap property.• each tree is min-heap ordered

• 0 or 1 binomial tree of order k

B4 B0B1

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3 18

54

Binomial Heap: Properties• Properties of N-node binomial heap.– Min key contained in root of B0, B1, . . . , Bk. (root list)

– Contains binomial tree Bi iff bi = 1 where bn b2b1b0 is binary representation of N.

– At most log2 N + 1 binomial trees.

– Height log2 N.

B4 B0B1

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3 18

N = 19

# trees = 3

height = 4

binary = 10011

55

Binomial Heap Operation 1

• Create a binomial heap

make-binomial-heap( )

H allocate-node( )

head[H] nil

return H

• Running time = O(1)

56

Binomial Heap Operation 2

• Minimum

• Running time = O(log n)

57

Binomial Heap Operation 3: Union

• Create heap H that is union of heaps H' and H''.

– "Mergeable heaps."

– Easy if H' and H'' are each order k binomial trees.

• connect roots of H' and H''

• choose smaller key to be root of H (Binomial-link)

H''55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

H'

58

Binomial Heap: Union

59


001 1

100 1+

011 1

11

1

1

0

1

19 + 7 = 26

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3 18

41

3328

15

25

7 12

+

60


55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3 18

41

3328

15

25

7 12

+

61


55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3

41

3328

15

25

7

+

12

18

18

12

62

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3

41

3328

15

25

7

+

12

18

25

377

3

18

12

18

12

63

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3

41

3328

15

25

7

12

+

18

25

377

3

41

28 33 25

3715 7

3

18

12

18

12

64

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3

41

3328

15

25

7

+

18

12

41

28 33 25

3715 7

3

12

18

25

377

3

41

28 33 25

3715 7

3

18

12

65

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

37

3

41

3328

15

25

7

+

18

12

41

28 33 25

3715 7

3

12

18

25

377

3

41

28 33 25

3715 7

3

55

45 32

30

24

23 22

50

48 31 17

448 29 10

6

18

12

68


• Create heap H that is union of heaps H' and H''.

– Analogous to binary addition.

• Running time = O(log N)

– Proportional to number of trees in root lists 2( log2 N + 1).

001 1

100 1+

011 1

11

1

1

0

1

19 + 7 = 26

69


• Running time = O(log N)

– Binomial-heap-merge: merge two sorted linked lists

• Proportional to number of trees in root lists 2( log2 N + 1).

– while loop (line 9- line 21)

• Each iteration executes one binomial-link O(1)

• Proportional to number of trees in root lists 2( log2 N + 1).

001 1

100 1+

011 1

11

1

1

0

1

19 + 7 = 26

70

Binomial Heap Operation 3:

Extract-min• Delete node with minimum key in binomial heap H.– Find root x with min key in root list of H, and delete

– H' broken binomial trees

– H Union(H', H)

• Running time. O(log N)

55

45 32

30

24

23 22

50

48 31 17

37

6 18

448 29 10

H

H'

73

Binomial Heap: Extract-min•Running time. O(log N)– 1. Find the root with minimum key from root list: O(# of trees) = O(log n)

– 2. Make-binomial-heap: O(1)

– 3. Reverse the order of the linked list of x’s children: O(degree(x)) = O(log n)

– 4. Binomial-heap-union: O(log n)

55

45 32

30

24

23 22

50

48 31 17

37

6 18

448 29 10

H

H'

74

Binomial Heap Operation 5: Insert

• Insert a new node x into binomial heap H.

– H' MakeHeap(x)

– H Union(H', H)


3

37

6 18

55

45 32

30

24

23 22

50

48 31 17

448 29 10

H

x

H'

75

Binomial Heap Operation 5: Insert

76

Binomial Heap: Sequence of Inserts• Insert a new node x into binomial heap H.

– If N = .......0, then only 1 steps.

– If N = ......01, then only 2 steps.

– If N = .....011, then only 3 steps.

– If N = ....0111, then only 4 steps.

• Inserting 1 item can take (log N) time.– If N = 11...111, then log2 N steps.

• But, inserting sequence of N items takes O(N) time!

– (N/2)(1) + (N/4)(2) + (N/8)(3) + . . . 2N

– Amortized analysis.

– Basis for getting most operationsdown to constant time.

50

48 31 17

4429 10

3

37

6 x

22

1

22

21

1

NN

N

nn

Nn

77

3

37

6 18

55

x 32

30

24

23 22

50

48 31 17

448 29 10

H

Binomial Heap Operation 6:

Decrease Key• Decrease key of node x in binomial heap H.– Suppose x is in binomial tree Bk.

– Bubble node x up the tree if x is too small.


– Proportional to depth of node x log2 N .

depth = 3

80

Binomial Heap Operation 7: Delete

• Delete node x in binomial heap H.

– Decrease key of x to -.

– Delete min.


81

Binomial Heap Operation 7: Delete

82

Questions?

design and analysis of algorithms -...

Documents