1

Theory of Computation

計算理論

2

Instructor: 顏嗣鈞E-mail: yen@ee.ntu.edu.tw

Web: http://www.ee.ntu.edu.tw/~yen

Time: 2:20-5:10 PM, Tuesday

Place: BL 112

Office hours: by appointment

Class web page: http://www.ee.ntu.edu.tw/~yen/courses/TOC-2006.htm 

3

• :

Introduction to Automata Theory, Languages, and Computation

John E. Hopcroft, Rajeev Motwani,

Jeffrey D. Ullman,

(2nd Ed. Addison-Wesley, 2001)

textbook

4

1st Edition

Introduction to Automata Theory, Languages, and Computation John E. Hopcroft, Jeffrey D. Ullman,

(Addison-Wesley, 1979)

5

Grading HW : 0-20%

Midterm exam.: 35-45%

Final exam.: 45-55%

6

Why Study Automata Theory?

Finite automata are a useful model for important kinds of hardware and software:

• Software for designing and checking digital circuits.

• Lexical analyzer of compilers. • Finding words and patterns in large bodies of

text, e.g. in web pages.• Verification of systems with finite number of

states, e.g. communication protocols.

7

Why Study Automata Theory? (2)The study of Finite Automata and Formal

Languages are intimately connected. Methods for specifying formal languages are very important in many areas of CS, e.g.:

• Context Free Grammars are very useful when designing software that processes data with recursive structure, like the parser in a compiler.

• Regular Expressions are very useful for specifying lexical aspects of programming languages and search patterns.

8

Why Study Automata Theory? (3)

Automata are essential for the study of the limits of computation. Two issues:

• What can a computer do at all? (Decidability)

• What can a computer do efficiently? (Intractability)

9

Applications

Theoretical Computer ScienceAutomata Theory, Formal Languages,

Computability, Complexity …

Compile

r

Prog. languages

Comm

. protocols

circuits

Pattern recognition

Supervisory control

Quantum

computing

Computer-

Aided Verification ...

10

Aims of the Course• To familiarize you with key Computer Science

concepts in central areas like- Automata Theory- Formal Languages- Models of Computation- Complexity Theory

• To equip you with tools with wide applicability in the fields of CS and EE, e.g. for- Complier Construction- Text Processing- XML

11

Fundamental Theme

• What are the capabilities and limitations of computers and computer programs?– What can we do with

computers/programs?– Are there things we cannot do with

computers/programs?

12

Studying the Theme

• How do we prove something CAN be done by SOME program?

• How do we prove something CANNOT be done by ANY program?

13

Example: The Halting Problem (1)

Consider the following program. Does it terminate for all values of n 1?

while (n > 1) {if even(n) {

n = n / 2;} else {

n = n * 3 + 1;}

}

14

Example: The Halting Problem (2)Not as easy to answer as it might first seem.Say we start with n = 7, for example:

7, 22, 11, 34, 17, 52, 26, 13, 40, 20, 10, 5,16, 8, 4, 2, 1

In fact, for all numbers that have been tried(a lot!), it does terminate . . .

. . . but in general?

15

Example: The Halting Problem (3)Then the following important undecidability

result should perhaps not come as a total surprise:

It is impossible to write a program that decides if another, arbitrary, program terminates (halts) or not.

What might be surprising is that it is possible to prove such a result. This was first done by the British mathematician Alan Turing.

16

Our focus

Automata Computability

Complexity

17

Topics1.  Finite automata, Regular languages, Regular

grammars: deterministic vs. nondeterministic, one-way vs. two-way finite automata, minimization, pumping lemma for regular sets, closure properties.

2.  Pushdown automata, Context-free languages, Context-free grammars: deterministic vs. nondeterministic, one-way vs. two-way PDAs, reversal bounded PDAs, linear grammars, counter machines, pumping lemma for CFLs, Chomsky normal form, Greibach normal form, closure properties.

3.

18

Topics (cont’d)3. Linear bounded automata, Context-

sensitive languages, Context-sensitive grammars.

4. Turing machines, Recursively enumerable sets, Type 0 grammars: variants of Turing machines, halting problem, undecidability, Post correspondence problem, valid and invalid computations of TMs.

 

19

Topics (cont’d)5. Basic recursive function theory 6. Basic complexity theory: Various

resource bounded complexity classes, including NLOGSPACE, P, NP, PSPACE, EXPTIME, and many more. reducibility, completeness.

7. Advanced topics: Tree Automata, quantum automata, probabilistic automata, interactive proof systems, oracle computations, cryptography.

20

Who should take this course?

YOU

21

Languages

The terms language and word are used in a strict technical sense in this course:

A language is a set of words. A word is a sequence (or string) of

symbols. (or ) denotes the empty word, the

sequence of zero symbols.

22

Symbols and Alphabets• What is a symbol, then?• Anything, but it has to come from an

alphabet which is a finite set.• A common (and important) instance is

= {0, 1}. , the empty word, is never an symbol

of an alphabet.

23

Computation

CPU memory

24

CPUinput memory

output memory

Program memory

temporary memory

25

CPUinput memory

output memoryProgram memory

temporary memory

3)( xxf

compute xx

compute xx 2

Example:

26

CPU

input memory

output memoryProgram memory

temporary memory

3)( xxf

compute xx

compute xx 2

2x

27

CPU

input memory

output memoryProgram memory

temporary memory 3)( xxf

compute xx

compute xx 2

2x

42*2 z82*)( zxf

28

CPU

input memory

output memoryProgram memory

temporary memory 3)( xxf

compute xx

compute xx 2

2x

42*2 z82*)( zxf

8)( xf

29

Automaton

CPUinput memory

output memory

Program memory

temporary memory

Automaton

30

Different Kinds of AutomataAutomata are distinguished by the temporary memory

• Finite Automata: no temporary memory

• Pushdown Automata: stack

• Turing Machines: random access memory

31

input memory

output memory

temporary memory

FiniteAutomaton

Finite Automaton

Example: Vending Machines (small computing power)

32

input memory

output memory

Stack

PushdownAutomaton

Pushdown Automaton

Example: Compilers for Programming Languages (medium computing power)

Push, Pop

33

input memory

output memory

Random Access Memory

TuringMachine

Turing Machine

Examples: Any Algorithm (highest computing power)

34

Finite Automata

PushdownAutomata

TuringMachine

Power of Automata

Less power More powerSolve more computational problems

35

Mathematical Preliminaries

36

Mathematical Preliminaries

• Sets • Functions• Relations• Graphs• Proof Techniques

37

}3,2,1{AA set is a collection of elements

SETS

},,,{ airplanebicyclebustrainB

We writeA1

Bship

38

Set Representations

C = { a, b, c, d, e, f, g, h, i, j, k }

C = { a, b, …, k }

S = { 2, 4, 6, … }

S = { j : j > 0, and j = 2k for some k>0 }

S = { j : j is nonnegative and even }

finite set

infinite set

39

A = { 1, 2, 3, 4, 5 }

Universal Set: all possible elements U = { 1 , … , 10 }

1 2 34 5

AU

67

8910

40

Set OperationsA = { 1, 2, 3 } B = { 2, 3, 4, 5}

• Union A U B = { 1, 2, 3, 4, 5 } • Intersection

A B = { 2, 3 }• Difference

A - B = { 1 }B - A = { 4, 5 }

U

A B

A-B

41

A

• ComplementUniversal set = {1, …, 7} A = { 1, 2, 3 } A = { 4, 5, 6, 7}

1 2 34

56

7A

A = A

42

024

61

35

7even

{ even integers } = { odd integers }

odd

Integers

43

DeMorgan’s Laws

A U B = A BU

A B = A U BU

44

Empty, Null Set:= { }

S U = S

S =

S - = S

- S =

U = Universal Set

45

SubsetA = { 1, 2, 3} B = { 1, 2, 3, 4, 5 }

A B

U

Proper Subset: A B

UAB

46

Disjoint SetsA = { 1, 2, 3 } B = { 5, 6}

A B = UA B

47

Set Cardinality• For finite setsA = { 2, 5, 7 }

|A| = 3

48

PowersetsA powerset is a set of sets

Powerset of S = the set of all the subsets of S

S = { a, b, c }

2S = { , {a}, {b}, {c}, {a, b}, {a, c}, {b, c}, {a, b, c} }

Observation: | 2S | = 2|S| ( 8 = 23 )

49

Cartesian ProductA = { 2, 4 } B = { 2, 3, 5 }

A X B = { (2, 2), (2, 3), (2, 5), ( 4, 2), (4, 3), (4, 5) }

|A X B| = |A| |B|

Generalizes to more than two sets

A X B X … X Z

50

FUNCTIONSdomain

123

ab

c

range

f : A -> B

A B

If A = domain then f is a total function otherwise f is a partial function

f(1) = a

51

RELATIONS R = {(x1, y1), (x2, y2), (x3, y3), …}

xi R yi

e. g. if R = ‘>’: 2 > 1, 3 > 2, 3 > 1

In relations xi can be repeated

52

Equivalence Relations• Reflexive: x R x• Symmetric: x R y y R x• Transitive: x R y and y R z x R z

Example: R = ‘=‘ • x = x• x = y y = x• x = y and y = z x = z

53

Equivalence ClassesFor equivalence relation R equivalence class of x = {y : x R y}

Example: R = { (1, 1), (2, 2), (1, 2), (2, 1), (3, 3), (4, 4), (3, 4), (4, 3) }

Equivalence class of 1 = {1, 2}Equivalence class of 3 = {3, 4}

54

GRAPHSA directed graph

• Nodes (Vertices) V = { a, b, c, d, e }• Edges E = { (a,b), (b,c), (b,e),(c,a), (c,e), (d,c), (e,b), (e,d) }

node

edge

ab

c

d

e

55

Labeled Graph

ab

c

d

e

1 35 6

262

56

Walk

ab

c

d

e

Walk is a sequence of adjacent edges (e, d), (d, c), (c, a)

57

Path

ab

c

d

e

Path is a walk where no edge is repeated

Simple path: no node is repeated

58

Cycle

ab

c

d

e

123

Cycle: a walk from a node (base) to itself

Simple cycle: only the base node is repeated

base

59

Euler Tour

ab

c

d

e 123

4 5 678 base

A cycle that contains each edge once

60

Hamiltonian Cycle

ab

c

d

e 123

45 base

A simple cycle that contains all nodes

61

Treesroot

leaf

parent

child

Trees have no cycles

62

root

leaf

Level 0

Level 1

Level 2

Level 3

Height 3

63

Binary Trees

64

PROOF TECHNIQUES

• Proof by induction

• Proof by contradiction

65

Induction

We have statements P1, P2, P3, …

If we know• for some b that P1, P2, …, Pb are true• for any k >= b that

P1, P2, …, Pk imply Pk+1

Then Every Pi is true

66

Proof by Induction• Inductive basis

Find P1, P2, …, Pb which are true

• Inductive hypothesisLet’s assume P1, P2, …, Pk are true, for any k >= b

• Inductive stepShow that Pk+1 is true

67

ExampleTheorem: A binary tree of height n has at most 2n leaves.Proof by induction: let L(i) be the number of leaves at level i

L(0) = 1L(1) = 2L(2) = 4

L(3) = 8

68

We want to show: L(i) <= 2i

• Inductive basis L(0) = 1 (the root node)

• Inductive hypothesisLet’s assume L(i) <= 2i for all i = 0, 1, …, k

• Induction stepwe need to show that L(k + 1) <= 2k+1

69

Induction Step

From Inductive hypothesis: L(k) <= 2k

Levelk

k+1

70

L(k) <= 2kLevel

k

k+1

L(k+1) <= 2 * L(k) <= 2 * 2k = 2k+1

Induction Step

71

RemarkRecursion is another thing

Example of recursive function:

f(n) = f(n-1) + f(n-2)

f(0) = 1, f(1) = 1

72

Proof by Contradiction

We want to prove that a statement P is true

• we assume that P is false• then we arrive at an incorrect conclusion• therefore, statement P must be true

73

ExampleTheorem: is not rational

Proof:Assume by contradiction that it is rational = n/m n and m have no common factors

We will show that this is impossible

2

2

74

= n/m 2 m2 = n2

Therefore, n2 is even n is evenn = 2 k

2 m2 = 4k2 m2 = 2k2m is evenm = 2 p

Thus, m and n have common factor 2

Contradiction!

2