chapter 10

Chapter 10

P and NP

Classes of Algorithms

Logarithmic(all bases have the same growth rate)

(log n) (log (log n))

Poly-logarithmic (n log n) (n2 log n)• ...

Polynomial (exponent is constant) (1) – Sub-linear (n0.001) – Sub-linear (n0.5) – Sub-linear (n) - Linear (n2) (n3)• ... (n100)

Classes of Algorithms

Exponential

(bases make a big difference)

(2n) (3n) (nn)

Factorial

k cn < n! < nn

• where k is some constant, and

• c is a constant less than n

Exponential-Exponential

nnn

The Great Divide

• All Polynomial Algorithms

• includes – linear

– poly-log

– log

– constant

– sub-linear

• All Exponential Algorithms

• includes– factorial

– poly-poly

Problems vs. Algorithms

• A single problem, i.e., sorting integers, can have many algorithms with very different run times.– Bucketsort (n)– Permutationsort (n!)

• General sorting– Mergesort (n log n)– Permutationsort (n!)


• A specific algorithm’s runtime can vary depending on the input

• Quicksort (n log n) – on most unsorted lists (n2) – on pre-sorted or partially sorted lists

• So, what can we say about the sorting problem?


• The sorting problem facts:– Can be solved in (n) is special cases where

the input can be mapped to integers.– Can be solve in (n log n) in the general case

(mergesort)– The sorting problem is Polynomial because

there exists at least one Polynomial algorithm for solving the sorting problem.


• Some problems are intrinsically harder to solve than others.

• Some problems are intrinsically polynomial• Some can NOT be solved in polynomial

time– At least most people think so...– But its hard to prove that something can’t be

done

Example: 1960’s

• The Longest Common Subsequence Problem (LCS)– Fasted known algorithm can only solve the

problem in (2n) time.• Does there exist an optimal LCS algorithm that has a

polynomial runtime?– In 1960, most computer scientists were leaning

toward an answer of “no” i.e., there does NOT exist a polynomial algorithm.

Example: 1980’s

• In 1981, two guys Smith and Waterman used Dynamic Programming to solve the LCS problem in (n2)-time using (n2)memory.

• This discover and many other discoveries led computer scientists to ponder the following question:

• Can all problems be solved in Polynomial Time?– Maybe we just haven’t discovered all the good

algorithms yet.

P or NP

• NP – set of problems were the solution can be verified as correct (or optimal) in Polynomial Time but we have yet to solve them in Polynomial Time.

• P – set of problems that can be solved deterministically in Polynomial Time

NP P

Polynomial Problems

• Searching

• Sorting

• Minimum Spanning Tree

• LCS Problem

• Shortest Path Problem

Non-deterministically Polynomial Problems

• The graph isomorphism problem of determining whether two graphs can be drawn identically

• The traveling salesman problem, where we want to know if there is a route of some length that goes through all the nodes in a certain network

• The boolean satisfiability problem, where we want to know if a certain formula in propositional logic with boolean variables is satisfiable or not

• Knapsack problem• Bin Packing problem

NP (not deterministic)

Repeat x times where x = O(nc)Develop an answer in O(nc) time

Verify if answer is correct in O(nc) time (Yes or don’t know)

If its correct, hooray! You solved the problem, so stop!

NP P

NP (not deterministic)Example (the bin packing problem)Repeat x times where x = O(nc)

Arrange n item randomlyi = 1For j = 1 to n

if(bin[i].total_weight+item[j].weight < bin_capcity)add item[j] to bin[i]

elsei++; add item[j] to bin[i]

return iif (i < ceil(items.total_weight/bin_capacity))

You solved the problem, so stop!

NP (not deterministic)Example (the bin packing problem)

Repeat x times where x = O(nc)

Arrange n item randomly

i = 1

For j = 1 to n

if(bin[i].total_weight+item[j].weight < bin_capcity)

add item[j] to bin[i]

else

i++;

add item[j] to bin[i]

return i

if (i < ceil(items.total_weight/bin_capacity))

You solved the problem, so stop!

• This algorithm is polynomial

• This algorithm will always terminate (finite)

• When this algorithm returns an answer it is always optimal

• Because of its random nature this algorithm may return different output given the same input.

• Sometimes this algorithm does not return an answer

P (deterministic)

• Repeat x times where x = O(nc)– Develop an answer or solution in O(nc) time

– Verify if it is correct in O(nc) time (YES or No)

– If its correct, hooray! You solved the problem, so stop!

NP P

P (deterministic)

• Repeat x times where x = O(nc)– Develop an answer or solution in O(nc) time

– Verify if it is correct in O(nc) time (YES or No)

– If its correct, hooray! You solved the problem, so stop!

Most Problems in P have straight-forward algorithms that compute a correct or optimal solution.

The verification is done as the solution is developed.

Consider sorting: every comparison and swap corrects an inversion and get you closer to a correctly sorted list. The algorithms stops when there are no more inversions to fix

P is a subset of NP

• 6 out of 10 leading theoretical computer scientists believe this to be true.

NP P

P = NP

• 1 out of 10 leading theoretical computer scientists believe this to be true. Every problem that can be “verified” in Polynomial time can be solved in Polynomial time.

NP = P

P ? NP

• 3 out of 10 leading theoretical computer scientists aren’t sure, haven’t made up their minds yet, or they are too busy thinking about ninjas to care.

NP ? P

NP Problems• Given a problem with no known polynomial

algorithm, can we prove that it can NOT be solved in Polynomial time.

• So far, no such proof exists for any NP problem.– And there are hundreds of NP problems with no known

polynomial algorithm.

– There is a lot of evidence to support the fact that P does not equal NP, but no one can prove it.

NP-Complete

• P – problems with polynomial deterministic algorithms

• NP – problems with polynomial non-deterministic algorithms

• NP Complete – problems in NP that are very likely not in P.

NP P

NP-Complete

Footnote: Exponential Problems• Example: Given a chess board, determine if a

player can win. Output: Possible or impossible.

NP P

NP-Complete

EXP: Problems that can be verified in Exponential time

NP-Complete (TSP Example)

• Find the shortest path from A back to A that visits every vertex.

A

B

C

D

E

...

1 2

123

1

9

4

4

6

5

4 75

7

NP-Complete (TSP Example)

Non-deterministic Algorithm:• Generate a random permutation of the vertices and

compute total length of the tour (tour_length)• Sum up the N smallest edges (min_tour)• If the tour_length = min_tour

– Output optimal answer• Otherwise, don’t give an answer.

NP

• Given a new problem, if you can create a non-deterministic polynomial algorithm, then its in NP

NP

PNP-Complete

new problem

P

• IF you can create a deterministic polynomial algorithm, then its in P

NPP

NP-Completenew

problem

P or NP

• IF you can NOT create a deterministic polynomial algorithm, then you really can’t be sure of anything..

NPP

NP-Complete new problem

P or NP• So, there are many problems

in NP, • Some are clearly in P, • All the rest are on the edge,

i.e., we don’t know if a P algorithm will be discovered or not

• But there is convincing evidence that some of the problems on the edge are not in P and these are the NP-complete problems.

NP

P

NP-Complete

NP-Complete• NP-Complete started with several famous

problems with no known P algorithm: – Maximum Clique, – Graph Coloring, – 3SAT, – Traveling Salesman

• Each of the problems can be “polynomially” transformed into the others.

• The transformation algorithm is called a reduction,• The reduction process needs to be polynomial.

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete• New problems can be added to NP-Complete if the following

two conditions are satisfied:

1. The new problem can be reduced into one of the existing problems in NP-Complete

2. An existing NP-Complete problem can be reduced into the new problem

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem

NP-Complete• Why must we the transformation be bi-directional?

• Here the new problem can be reduced to the TSP problem.

• Thus, a TSP algorithm could be used to solve the new problem.

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem


• A reduction exists that can can transform the sorting problem to the TSP problem

• But, sorting is a Polynomial problem, so this alone isn’t enough to show that sorting is NP-Complete

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem


• Here the TSP problem could be reduced to the new problem

• Thus, any algorithm that solves the new problem could be used to solve the TSP problem.

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem


• First, there exists no such reduction to go from TSP to sorting, and this is why Sorting is not NP Complete.

• But if such a reduction existed than any algorithm for solving the new problem, could be used to solve all NP-Complete problems

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem


• But this direction is all that matters, right?

• Wrong, some NP-Complete problems can be reduced to problem that aren’t even in NP.

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem

This direction proves that the new problem has an NP Algorithm in the first place

NP-Complete• Once both reductions have been proven, the new problem gets

added to the set of NP-Complete problems.

• By the principle of transitivity, all NP-Complete problems can be reduce to the new problem and

• The new problem can be reduced to all NP-Complete problems

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

New Problem

The power of NP-Complete

• Adds mounting evidence that P != NP• More problems = more opportunity to find a

polynomial algorithm for an NP-Complete problem

• All you have to do is prove two reductions• One establishes that an NP algorithm exists• The other establishes equivalence (one

algorithm can solve all problems)

Proving that P = NP • All you have to do is find a problem with a known

polynomial algorithm, and• Develop a reduction from any of the NP-complete

problems (over 100).

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

Problem with known

Polynomial Algorithm

Polynomial reduction

Proving that P = NP • This should be really easy because there are...• Tons of problems with Polynomial algorithms, and• Tons of problems in NP-complete that could be reduced...

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

Problem with known



Proving that P = NP • This has never been done, which tell you

something...• Probably P != NP.

TSP 3SAT

Graph Coloring

Max Clique

NP-Complete

Problem with known



Halting Problem• Imagine if you had a function that

could generate a good solution for an NP-complete problem (in polynomial time).

• Imagine if you had function that could verify the optimality(in polynomial time).

• Put the algorithm in an infinite loop until an optimal solution was verified.

• Halting Problem: Will the algorithm every terminate?

• Harder Halting Problem: Will it terminate in polynomial time?

while (!optimal) {

a = generator(input);

if (is_optimal(a))

optimal = true;

}

Halting Problem

• If the answer generator is good enough, the algorithm might always stop.

• If the answer generator is really good, that algorithm might always stop in polynomial time.

Polynomial Halting Problem:• Given N, will the algorithm will finish in

O(Nk) time where k is a constant?

Halting Problem

• Not only is it impossible to verify or prove that a program will terminate in polynomial time, but...

• It is impossible to verify or prove that a program will even terminate (in the general case).

• The halting problem is interesting because it shows how difficult it is to prove that an NP-complete problem can NOT be solved in polynomial time.

Halting Problem

halt(p, i)• returns true if p describes a program that halts

when given as input the string i, and• returns false otherwise.

Interesting fact: It is impossible to create a halt(p,i) algorithm that will terminate for all programs.There are some programs where halt(p,i) will run forever and we can prove it.

Halting Problem

Important connection

If you can’t write an algorithm to determine if an arbitrary program halts then,

You can’t write an algorithm to determine if an arbitrary program halts in polynomial time.

Thus, there may be no way to prove that a problem can NOT be solved in polynomial time.

Something to ponder...• Imagine if you feed halt(p,i) the following program...

while (!optimal) {

a = generator(i);

if (is_optimal(a))

optimal = true;

} ....and its been running for 10 years?• It could still be polynomial (imagine O(n10) for

n = 1,000) and halt(p,i) might be just about to finish and return true.

• But you will never know if halt(p,i) will eventually return an answer and you could be waiting forever

Halting Problem

Some background: • Any program can be encoded as a binary string.• Any input can be encoded as a binary string.• You could actually feed an algorithm its own

encoding as an input.– Lets call this algorithm self-cannibalism.

Halting Problem

Some background: • I could feed a program itself in the following

way... halt(p, p).• Halt-pee-pee is the self-cannibalistic halting

problem: Will a program halt if you feed it its own encoding as input?

• If we could truly implement the halt algorithm to return an answer for all input, then it should even work on p,p.

WTF

function wtf(string p)

if halt(p, p) == false

return true

else

loop forever

wft is a program that takes another program p encoded as a string. wft simply calls halt(p, p) and does the opposite.

Halting Problem

function wtf(string p)

if halt(p, p) == false

return true

else

loop forever

Here is why wft if so crazy. We could actually encode wft as a string and feed the string to wtf, making it a self-cannibalistic algorithm.

Halting Problem (Contradiction 1)

function wtf(string p) if halt(p, p) == false

return true else

loop forever

• If wtf stops and returns true, it must be because halt(p, p) returned false,

• but halt only returns false if p runs forever, and p is actually the encoding for wtf. This means that wtf should not have stopped if you feed it to itself.

Halting Problem (Contradiction 2)

function wtf(string p) if halt(p, p) == false

return true else

loop forever

• If wtf loops forever, it is because halt(p,p) returned true. • but halt only returns true if p will stop, and p is actually the

encoding of wtf. This means that wtf should have stopped if you feed it to itself.

Proof by reduction to the absurd

• The assumption that we’ve made is that halt(p,i) exists and returns an answer for all programs p.

• If this assumption is true then halt(p,p), i.e., self cannibalism should work.

• But what if you create a program called wtf(p) that actually calls halt(p,p). This alone is not a problem.

• However, if you feed the encoding of wtf into wtf, you create an absurd paradox.

Understanding the absurd paradox

• Consider a vast library that contains all the books in the world.

• Now consider the library’s catalogs.• A catalog is simply a book that lists the location of

other books.• Examples:

– The Catalog of all Biology books– The Catalog of all Chemistry books– The Catalog of all Science Related Catalogs


• Examples: – The Catalog of all Biology books– The Catalog of all Chemistry books– The Catalog of all Science Related Catalogs

• Clearly, some catalogs could actually include the location of other catalogs.


• Clearly, some catalogs could actually include the location of other catalogs.

• Now, consider the catalog of all catalogs (CoaC)

• Clearly, the CoaC lists itself.– This is similar to how table of contents also

includes the page number of the table of contents. Its also similar to algorithm self-cannibalism.


• Now, consider the catalogue of all catalogues that don't contain themselves.– This catalog is similar to the wtf program, so lets call it

the wtf catalog.

• The wtf catalog would include straightforward catalogs– Catalog of Bird books

– Catalog of Dog books

• The wtf catalog would NOT include the CoaC.


• Now, consider this question. Will the wtf catalog be listed in the wtf_catalog?

• This is similar to feeding the wtf program its own encoding

• Contradiction 1: If the wtf_catalog is listed inside the wtf_catalog, then its a self referential catalog so it can’t be listed.

• Remember the wtf_catalog list all catalogs that are not self-referencing.


• Contradiction 2: If the wtf_catalog is NOT listed inside the wtf_catalog, then its NOT a self-referential catalog so it must be listed in the wtf_catalog.

• Remember to be a self-referential catalog, you have to be listed within your pages.


• These two contractions tell you that the wtf catalog can not exist unless you allow for inconsistency.

• Thus, wtf itself doesn’t have to follow the rules.

• It can be self-referential but still be listed in the catalog of all non-self-referential catalogs.


• In world of libraries, books and catalogs it is probably OK to not have consistency for one special case.

• But in the world of math and algorithms, one instance of inconsistency, undermines the entire system.

• Thus, statements can be proved and disproved at the same time.


• For the library example, the way to resolve the inconsistency is to simply agree that wtf_catalog can not exist according to its own rules.

• For the halting problem, you can also agree that the wtf program can’t exist according to its own rules.

• But the wtf program does exist, feel free to implement it after class.

• The real show stopper is that the wtf program is perfectly consistent as long as you assume halt(p,p) may never stop and return an answer.


• The real show stopper is that the wtf program is perfectly consistent as long as you assume halt may never stop and return an answer.

• As long as there are some problems and inputs that force halt into an infinite loop, then the wtf program can exist without being inconsistent.

Summary

• We have no choice but to conclude that halt will not always return an answer for all problems and all inputs.

• halt(p,i) will sometimes loop infinitely.• What sucks is that if halt(p,i) was in fact a finite

algorithm, then perhaps we could create halt_in_polynomial(p,i), which could help us unravel the P = NP dilemma

• But since halt(p,i) can’t be a finite algorithm, there is no way halt_in_polynomial(p,i) could be a finite algorithm, because is a much harder problem.

Summary

• The halting problem serves as further evidence that the P = NP problem may never be solved.

• This all centers around Gödel ’s proof that no system, whether mathematical or algorithmic, can be both consistent and complete.

• The nature of Algorithms favor consistency, so we are left with an incomplete system.

• Incomplete means that the system doesn’t provide all the tools to prove things that might actually be true.

Conclusion

• To solve certain problems efficiently, we have to invent an entirely new system for developing algorithms.

• That system may be based on quantum physics, where (in theory) information can be passed instantly and NN computations could be executed in parallel on one quantum computer.

• I don’t think its going to happen, but 1000 years ago no one thought that you could have sorted a thousand items in one second.

chapter 10

Documents