algorithms,,graph,theory,,and,, linear,equa9ons,in,laplacians, · 2010-08-15 ·...

Algorithms, Graph Theory, and Linear Equa9ons in Laplacians

Daniel A. Spielman Yale University

Shang-‐Hua Teng 滕尚華

Pavel Cur9s

David Harbater UPenn

Carter Fussell TPS

Todd Knoblock

CTY

Serge Lang 1927-‐2005

Gian-‐Carlo Rota 1932-‐1999

Algorithms, Graph Theory, and Linear Equa9ons in Laplacians

Daniel A. Spielman Yale University

Solving Linear Equa9ons Ax = b, Quickly

Goal: In 9me linear in the number of non-‐zeros entries of A

Special case: A is the Laplacian Matrix of a Graph

Solving Linear Equa9ons Ax = b, Quickly

Goal: In 9me linear in the number of non-‐zeros entries of A

Special case: A is the Laplacian Matrix of a Graph

1. Why 2.  Classic Approaches 3.  Recent Developments 4.  Connec9ons

Graphs

Set of ver9ces V. Set of edges E of pairs u,v ⊆ V

Graphs

Erdös

Lovász Alon

Karp

8

1

2 7 5 3

4 10 9 6

Set of ver9ces V. Set of edges E of pairs u,v ⊆ V

Laplacian Quadra9c Form of G = (V,E)

For x : V → IR

xTLGx =

(u,v)∈E

(x (u)− x (v))2


For x : V → IR

−1−3 01

3

x :

xTLGx =

(u,v)∈E

(x (u)− x (v))2


For x : V → IR

−1−3 0x :

xTLGx = 15

22 1212

32

xTLGx =

(u,v)∈E

(x (u)− x (v))2

1

3

Laplacian Quadra9c Form for Weighted Graph

xTLGx =

(u,v)∈E

w(u,v) (x (u)− x (v))2

G = (V,E,w)

w : E → IR+ assigns a posi9ve weight to every edge

Matrix LG is posi9ve semi-‐definite nullspace spanned by const vector, if connected

Laplacian Matrix of a Weighted Graph

LG(u, v) =

−w(u, v) if (u, v) ∈ E

d(u) if u = v

0 otherwise

4 -1 0 -1 -2! -1 4 -3 0 0! 0 -3 4 -1 0! -1 0 -1 2 0! -2 0 0 0 2!

1 2

3 4

5 1!

1!

2!

1!

3!

d(u) =

(v,u)∈E w(u, v)

the weighted degree of u

Laplacian Matrix of a Weighted Graph

LG(u, v) =

−w(u, v) if (u, v) ∈ E

d(u) if u = v

0 otherwise

4 -1 0 -1 -2! -1 4 -3 0 0! 0 -3 4 -1 0! -1 0 -1 2 0! -2 0 0 0 2!

1 2

3 4

5 1!

1!

2!

1!

3!

d(u) =

(v,u)∈E w(u, v)

the weighted degree of u combinatorial degree is # of aaached edges

3

2

2

2

1

Networks of Resistors

Ohm’s laws gives

In general, with w(u,v) = 1/r(u,v)

Minimize dissipated energy

i = v/r

i = LGv

vTLGv

1V

0V

1V

0V

0.5V

0.5V

0.625V 0.375V

By solving Laplacian

1V

0V

Networks of Resistors

Ohm’s laws gives

In general, with w(u,v) = 1/r(u,v)

Minimize dissipated energy

i = v/r

i = LGv

vTLGv

Learning on Graphs

Infer values of a func9on at all ver9ces from known values at a few ver9ces.

Minimize xTLGx =

(u,v)∈E

w(u,v) (x (u)− x (v))2

Subject to known values

0

1

Learning on Graphs

0

1 0.5

0.5

0.625 0.375

By solving Laplacian

Infer values of a func9on at all ver9ces from known values at a few ver9ces.

Minimize xTLGx =

(u,v)∈E

w(u,v) (x (u)− x (v))2

Subject to known values

Spectral Graph Theory

Combinatorial proper9es of G are revealed by eigenvalues and eigenvectors of LG

Compute the most important ones by solving equa9ons in the Laplacian.

Solving Linear Programs in Op9miza9on

Interior Point Methods for Linear Programming: network flow problems Laplacian systems

Numerical solu9on of Ellip9c PDEs

Finite Element Method

How to Solve Linear Equa9ons Quickly

Fast when graph is simple, by elimina9on.

Fast when graph is complicated*, by Conjugate Gradient (Hestenes ‘51, S9efel ‘52)

Cholesky Factoriza9on of Laplacians

Also known as Y-‐Δ

When eliminate a vertex, connect its neighbors.

3 -1 0 -1 -1! -1 2 -1 0 0! 0 -1 2 -1 0! -1 0 -1 2 0! -1 0 0 0 1!

1 2

3 4

5 1!

1!

1!

1!

1!




3 -1 0 -1 -1! -1 2 -1 0 0! 0 -1 2 -1 0! -1 0 -1 2 0! -1 0 0 0 1!

1 2

3 4

5 1!

1!

1!

1!

1!.33!

.33!

.33!

3 0 0 0 0! 0 1.67 -1.00 -0.33 -0.33! 0 -1.00 2.00 -1.00 0! 0 -0.33 -1.00 1.67 -0.33! 0 -0.33 0 -0.33 0.67!




3 -1 0 -1 -1! -1 2 -1 0 0! 0 -1 2 -1 0! -1 0 -1 2 0! -1 0 0 0 1!

1 2

3 4

5

1!

1!.33!

.33!

.33!

3 0 0 0 0! 0 1.67 -1.00 -0.33 -0.33! 0 -1.00 2.00 -1.00 0! 0 -0.33 -1.00 1.67 -0.33! 0 -0.33 0 -0.33 0.67!

3 0 0 0 0! 0 1.67 -1.00 -0.33 -0.33! 0 -1.00 2.00 -1.00 0! 0 -0.33 -1.00 1.67 -0.33! 0 -0.33 0 -0.33 0.67!

3 -1 0 -1 -1! -1 2 -1 0 0! 0 -1 2 -1 0! -1 0 -1 2 0! -1 0 0 0 1!

3 0 0 0 0! 0 1.67 0 0 0! 0 0 1.4 -1.2 -0.2! 0 0 -1.2 1.6 -0.4! 0 0 -0.2 -0.4 0.6!

1 2

3 4

5 1!

1!

1!

1!

1!

1 2

3 4

5 .33!

.33!

1!

1!.33!

1 2

3 4

5 .2!

1.2!

.4!

1 0 0 0 0! 0 2 -1 0 -1! 0 -1 2 -1 0! 0 0 -1 2 -1! 0 -1 0 -1 2!

1 -1 0 0 0! -1 3 -1 0 -1! 0 -1 2 -1 0! 0 0 -1 2 -1! 0 -1 0 -1 2!

1 0 0 0 0! 0 2 0 0 0! 0 0 1.5 -1 -0.5! 0 0 -1.0 2 -1.0! 0 0 -0.5 -1 1.5!

2 3

4 5

1 1!

1!

1!

1!

1!

2 3

4 5

1 1!

1!

1!

1!

2 3

4 5

1

1!

1!

0.5!

The order maaers

Complexity of Cholesky Factoriza9on

#ops ~ Σv (degree of v when eliminate)2

Tree (connected, no cycles)

#ops ~ O(|V|)



#ops ~ O(|V|) Tree (connected, no cycles)



Tree #ops ~ O(|V|)



Tree #ops ~ O(|V|)

Planar #ops ~ O(|V|3/2) Lipton-‐Rose-‐Tarjan ‘79



Tree #ops ~ O(|V|)

Planar #ops ~ O(|V|3/2) Lipton-‐Rose-‐Tarjan ‘79

Expander like random, but O(|V|) edges

#ops ≳ Ω(|V|3) Lipton-‐Rose-‐Tarjan ‘79

Conductance and Cholesky Factoriza9on

Cholesky slow when conductance high Cholesky fast when low for G and all subgraphs

For S ⊂ V

ΦG = minS⊂V Φ(S)

Φ(S) = # edges leaving Ssum degrees on smaller side, S or V − S

S

Conductance

S

Conductance

Φ(S) = 3/5

S

Conductance

S

Cheeger’s Inequality and the Conjugate Gradient

Cheeger’s inequality (degree-‐d unwted case)

12λ2d ≤ ΦG ≤

2λ2

d

= second-‐smallest eigenvalue of LG ~ d/mixing 9me of random walk

λ2

Cheeger’s Inequality and the Conjugate Gradient

Cheeger’s inequality (degree-‐d unwted case)

12λ2d ≤ ΦG ≤

2λ2

d

= second-‐smallest eigenvalue of LG ~ d/mixing 9me of random walk

λ2

Conjugate Gradient finds ∊ -‐approx solu9on to LG x = b

in mults by LG O(d/λ2 log −1)

in ops O(|E|

d/λ2 log −1)

Fast solu9on of linear equa9ons

CG fast when conductance high.

Elimina9on fast when low for G and all subgraphs.




Planar graphs

Want speed of extremes in the middle




Planar graphs

Want speed of extremes in the middle

Not all graphs fit into these categories!

Precondi9oned Conjugate Gradient

Solve LG x = b by

Approxima9ng LG by LH (the precondi9oner)

In each itera9on solve a system in LH mul9ply a vector by LG

∊ -‐approx solu9on ater O(

κ(LG, LH) log −1) itera9ons

Precondi9oned Conjugate Gradient

Solve LG x = b by

Approxima9ng LG by LH (the precondi9oner)

In each itera9on solve a system in LH mul9ply a vector by LG

∊ -‐approx solu9on ater O(

κ(LG, LH) log −1) itera9ons

rela6ve condi6on number

Inequali9es and Approxima9on

if is positive semi-definite, "i.e. for all x,

xTLHx xTLGx

LH LG LG − LH

Example: if H is a subgraph of G

xTLGx =

(u,v)∈E

w(u,v) (x (u)− x (v))2



κ(LG, LH) ≤ t

LH LG tLH

LH LG LG − LH

if

iff for some c cLH LG ctLH

xTLHx xTLGx



κ(LG, LH) ≤ t

LH LG tLH

LH LG LG − LH

if

iff for some c cLH LG ctLH

Call H a t-‐approx of G if κ(LG, LH) ≤ t

xTLHx xTLGx

Other defini9ons of the condi9on number

κ(LG, LH) =

max

x∈Span(LH)

xTLGx

xTLHx

max

x∈Span(LG)

xTLHx

xTLGx

pseudo-inverse

min non-zero eigenvalue

κ(LG, LH) =λmax(LGL

+H)

λmin(LGL+H)

(Golds9ne, von Neumann ‘47)

Vaidya’s Subgraph Precondi9oners

Precondi9on G by a subgraph H

LH LG so just need t for which LG tLH

Easy to bound t if H is a spanning tree

And, easy to solve equa9ons in LH by elimina9on

H

The Stretch of Spanning Trees

Where

Boman-‐Hendrickson ‘01:

stT (G) =

(u,v)∈E

path-lengthT (u, v)

LG stG(T )LT


path-‐len 3

Where


stT (G) =

(u,v)∈E

path-lengthT (u, v)

LG stG(T )LT


path-‐len 5

Where


stT (G) =

(u,v)∈E

path-lengthT (u, v)

LG stG(T )LT


path-‐len 1

Where


stT (G) =

(u,v)∈E

path-lengthT (u, v)

LG stG(T )LT


In weighted case, measure resistances of paths

Where


stT (G) =

(u,v)∈E

path-lengthT (u, v)

LG stG(T )LT

Low-‐Stretch Spanning Trees

(Alon-‐Karp-‐Peleg-‐West ’91)

(Elkin-‐Emek-‐S-‐Teng ’04, Abraham-‐Bartal-‐Neiman ’08)

For every G there is a T with

where m = |E|

Solve linear systems in 9me O(m3/2 logm)

stT (G) ≤ m1+o(1)

stT (G) ≤ O(m logm log2 logm)

Low-‐Stretch Spanning Trees

(Alon-‐Karp-‐Peleg-‐West ’91)

(Elkin-‐Emek-‐S-‐Teng ’04, Abraham-‐Bartal-‐Neiman ’08)

For every G there is a T with

where m = |E|

If G is an expander

stT (G) ≤ m1+o(1)

stT (G) ≤ O(m logm log2 logm)

stT (G) ≥ Ω(m logm)

Expander Graphs

Infinite family of d-‐regular graphs (all degrees d) satsifying

Spectrally best are Ramanujan Graphs (Margulis ‘88, Lubotzky-‐Phillips-‐Sarnak ’88) all eigenvalues inside

λ2 ≥ const > 0

d± 2√d− 1

Amazing proper9es

Fundamental examples

Expanders Approximate Complete Graphs

Let G be the complete graph on n ver9ces (having all possible edges)

All non-‐zero eigenvalues of LG are n

for all x ⊥ 1, x = 1xTLGx = n

Expanders Approximate Complete Graphs

Let G be the complete graph on n ver9ces

Let H be a d-‐regular Ramanujan Expander

κ(LG, LH) ≤ d+ 2√d− 1

d− 2√d− 1

→ 1

for all x ⊥ 1, x = 1xTLGx = n

(d− 2√d− 1) ≤ xTLHx ≤ (d+ 2

√d− 1)

Sparsifica9on

Goal: find sparse approxima9on for every G

S-‐Teng ‘04: For every G is an H with O(n log7 n/2) edges and κ(LG, LH) ≤ 1 +

Sparsifica9on


Conductance high high (Cheeger) random sample good (Füredi-‐Komlós ‘81)

λ2

Conductance not high can split graph while removing few edges

Fast Graph Decomposi9on by local graph clustering

Given vertex of interest find nearby cluster, small , in 9me

Φ(S)

Fast Graph Decomposi9on by local graph clustering

S-‐Teng ’04: Lovász-‐Simonovits Andersen-‐Chung-‐Lang ‘06: PageRank Andersen-‐Peres ‘09: evoloving set Markov chain

Given vertex of interest find nearby cluster, small , in 9me

Φ(S)

Sparsifica9on



S-‐Srivastava ‘08: with edges proof by modern random matrix theory Rudelson’s concentra9on for random sums

O(n log n/2)

Sparsifica9on



S-‐Srivastava ‘08: with edges

Batson-‐S-‐Srivastava ‘09

dn edges and κ(LG, LH) ≤ d+ 1 + 2√d

d+ 1− 2√d

O(n log n/2)

Sparsifica9on by Linear Algebra

edges vectors

Given vectors s.t. v1, ..., vm ∈ IRn

e

vevTe = I

Find a small subset S ⊂ 1, ...,mand coefficients s.t. ce

||

e∈S

cevevTe − I|| ≤



e

vevTe = I

Find a small subset S ⊂ 1, ...,m

and coefficients s.t. ce ||

e∈S

cevevTe − I|| ≤

Rudelson ‘99 says can find if choose S at random*

* = In graphs, are effec9ve resistances

|S| ≤ O(n log n/2)

Pr [e] ∼ 1/ ve2


Batson-‐S-‐Srivastava: can find by greedy algorithm

with S9eltjes poten9al func9on

|S| ≤ 4n/2


e

vevTe = I

Find a small subset S ⊂ 1, ...,m

and coefficients s.t. ce ||

e∈S

cevevTe − I|| ≤

Rela9on to Kadison-‐Singer, Paving Conjecture

e

vevTe = I

Exists

Would be implied by the following strong version of Weaver’s conjecture KS’2

ve2 =n

m≤ α

S ⊂ 1, ...,m , |S| = m/2

e∈S

vevTe − 1

2I

≤ 1

4

Exists constant α s.t. for s.t. for v1, ..., vm ∈ IRn


Rudelson ‘99:

Batson-‐S-‐Srivastava ‘09: true, but with coefficients in sum

α = const/(log n)

e

vevTe = I

Exists

ve2 =n

m≤ α

S ⊂ 1, ...,m , |S| = m/2

e∈S

vevTe − 1

2I

≤ 1

4



Rudelson ‘99:

Batson-‐S-‐Srivastava ‘09: true, but with coefficients in sum

α = const/(log n)

e

vevTe = I

Exists

ve2 =n

m≤ α

S ⊂ 1, ...,m , |S| = m/2

e∈S

vevTe − 1

2I

≤ 1

4


S-‐Srivastava ‘10: Bourgain-‐Tzafriri Restricted Invertability

Sparsifica9on and Solving Linear Equa9ons

Can reduce any Laplacian to one with non-‐zero entries/edges.

S-‐Teng ‘04: Combine Low-‐Stretch Trees with Sparsifica9on to solve Laplacian systems in 9me

O(m logc n log −1)

m = |E| n = |V |

O(|V |)

Sparsifica9on and Solving Linear Equa9ons

S-‐Teng ‘04: Combine Low-‐Stretch Trees with Sparsifica9on to solve Laplacian systems in 9me

O(m logc n log −1)

m = |E| n = |V |

Kou9s-‐Miller-‐Peng ’10: 9me

Kolla-‐Makarychev-‐Saberi-‐Teng ’09: ater preprocessing

O(m log2 n log −1)

O(m log n log −1)

What’s next

Decomposi9ons of the iden9ty: Understanding the vectors we get from graphs the Ramanujan bound

Kadison-‐Singer?

Other families of linear equa9ons: from directed graphs from physical problems from op9miza9on problems

Solving Linear equa9ons as a primi9ve

Conclusion

It is all connected

algorithms,,graph,theory,,and,, linear,equa9ons,in,laplacians, · 2010-08-15 ·...

Documents