Manifold Learning on Probabilistic Graphical Models概率图上的流形学习

答辩人 : 邵元龙导师 : 鲍虎军 教授 & 何晓飞 教授

浙江大学 CAD&CG 国家重点实验室2010 年 3 月 5 日

Outline

Background & Motivation Function Learning v.s. Statistical Modeling

Manifold Regularized Variational Inference Algorithm Design & Examples In Depth Analysis Implementation Experimental Results

Function Learning Given data points , and a function space , find the optimal function , such that

Regularization is Important!!

N 1

,N

i i ix y

2*

1

arg min ,N

i if i

f V y f x f

F*f F

Statistical Modeling All quantities, no matter given or to be

estimated, are random variables. Then we model the joint distribution.

, ,,|

, ,

p dpp

p p d d

H V Θ θ θH V

H VV H h V Θ θ h θ

e.g. Gaussian Mixture Model

Difficulties

How many components are there? Should there be any “components” ?

Difficulties (continued)

What if data reside on a non-trivial manifold

Efforts towards Non-Parametric, but …

What we want…

Review GMM

Function Learning embedded. :f X Z

Problem Formulation

What to regularize? Where to regularize?

Manifold Learning

Manifold Assumption Y changes smoothly with X, and we have

so should be small over manifold Minimizing it over the manifold,

f x x f x

f xx

f x

2min

xf dx

MM

Manifold Regularization

2

2

2,

min

1x

ij i ji j

f dx

S f x f xN

S

MM

Manifold Regularization

2

2

2,

2

2,

min

1

1

x

iji

i

j

ij j

i

j

j

i

f dx

f x f xSN

S yN

y

S

S

MM

Transductive Learning

Problem Formulation

What to regularize? Where to regularize?

Variational Inference For , define , a var. dist.

Approximate the true posterior with it

by minimizing the KL divergence

H H q H

|H

q q H p

H

H H V

* arg min || |q

q KL q pH

H H H V

Manifold Regularized Variational Inference

* arg minq

q H

H F

2

,

|| |

,ij i ji j

KL q p

S d q Z q Z

H H V

S

F

How to Optimize?

* arg minH

H q H

q H

H

HF

2

,

|| |

,ij i ji j

KL q p

S d q Z q Z

H H V

S

F

Optimization Algorithm

* arg mini

iii

q ZZ Zq Z F

2

,: ,

|| | , 2 ,ii iZ Zi i Z ij i j

j i j j i

KL q Z p Z S d q Z q Z

MB S

F

1

*

for ,...,

arg min || |

N

Hq H

H Z Z

q H KL q H p H

MB

An Illustration

0.2 2 200

Works Done

Example Distribution Types Convergence Proof Convexity Analysis (More TODO) Computational Complexity Numerical Stability A Flexible Inference Engine

YASIE (Yet Another Statistical Inference Engine)

Interface Design Inference Scheduling Type-Free Mixture Model Design Issues (e.g. Balance of Memory & Comp.

Time)

Experiments

Data Clustering Gaussian Mixture Model

Image Annotation Link Mixture of Unigram

Image Annotation Model Link Mixture of Unigram

Image Similarity Graph

“?” should be something like “Barcelona”

?

Image Annotation Performances

Image Annotation Examples

Any Question? 实验室的老师们：鲍虎军老师，何晓飞

老师，蔡登老师，刘新国老师，章国锋老师，黄劲老师……

师兄师弟师妹们：董子龙，姜翰青，周源，张驰原，林斌斌，薛维，瞿新泉，姚冠红……感谢你们一直以来给我的帮助！