enforcing convexity for improved alignment with constrained local models authors: yang wang, simon...

Enforcing Convexity for Improved Alignment with Constrained Local Models

Authors: Yang Wang,Simon Lucey,Jeffrey F. Cohn

讲解人 : 赵小伟文章出处 :CVPR’08

提纲作者信息文章信息背景知识 /(Constrained Local Models) 拟解决的问题与采用的思路实验结论 Demos

第一作者 Yang Wang

Research scientist at Siemens Robotics Institute, Carnegie Mellon University Ph.D. in Computer Science, August 2000 - December 2006

Stony Brook University, New York, USA B.S. in Computer Science, September 1993 - July 1998

Tsinghua University, Beijing, China

Research Areas Computer Vision, Graphics, Medical Image Analysis, Biometrics,

Machine Learning, Computer Animation, and Augmented Reality

Publication PAMI(07, 09) , IJCV(08), IVC(08, 09, 10), ECCV(02, 08),

CVPR(04, 06, 07, 08, 10), ICCV(05, 07, 09), FG(08)

Homepage http://www.cs.cmu.edu/~wangy/

第二作者 Simon Lucey

PhD Student, Universitat Pompeu Fabra Research Interest

I am passionate about gaining a deeper understanding of perception, learning and intelligence. My practical interests are in analyzing faces, biometrics and human event recognition. From an academic perspective I am extremely interested in computer vision, machine learning and how these evolving topics relate to deeper questions concerning Artificial Intelligence (AI).

Publication PAMI’10 , IVC’10, IJCV’08, PRL’07, Multimedia’05, ICCV, CVPR

Homepage http://www.cs.cmu.edu/~slucey/Main.html

第三作者 Jeffrey F. Cohn

Jeffrey Cohn is Professor of Psychology at the University of Pittsburgh and Adjunct Faculty at the Robotics Institute at Carnegie Mellon University.

Research Interest He has led interdisciplinary and inter-institutional efforts to develop

advanced methods of automatic analysis of facial expression and prosody; and applied those tools to research in human emotion, social development, non-verbal communication, psychopathology, and biomedicine.

Database Cohn-Kanade AU-Coded Facial Expression Database. CK Cohn-Kanade Expanded. CK+ CMU MultiPie.

Homepage http://www.pitt.edu/~jeffcohn/

文章信息文章出处

CVPR 2008

相关文献 [7] D. Cristinacce and T.F. Cootes. Feature detection and

tracking with constrained local models. In BMVC, pages 929-938, 2006

[3] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unifying framework: Part 1: The quantity approximated, the warp update rule, and the gradient descent approximation. IJCV, 2004.

[19] Y. Wang, S. Lucey, and J. Cohn. Non-rigid object alignment with a mismatch template based on exhaustive local search. In IEEE Workshop on Non-rigid Registration and Tracking through Learning, 2007.

Abstract Constrained local models (CLMs) have recently

demonstrated good performance in non-rigid object alignment/tracking in comparison to leading holistic approaches (e.g., AAMs). A major problem hindering the development of CLMs further, for non-rigid object alignment/tracking, is how to jointly optimize the global warp update across all local search responses. Previous methods have either used general purpose optimizers (e.g., simplex methods) or graph based optimization techniques. Unfortunately, problems exist with both these approaches when applied to CLMs.

In this paper, we propose a new approach for optimizing the global warp update in an efficient manner by enforcing convexity at each local patch response surface.

Abstract Furthermore, we show that the classic Lucas-Kanade

approach to gradient descent image alignment can be viewed as a special case of our proposed framework.

Finally, we demonstrate that our approach receives improved performance for the task of non-rigid face alignment/tracking on the MultiPIE database and the UNBC-McMaster archive.

摘要与基于全局的方法相比 ( 例如 AAM) ，带有局部约束的模型 (CLMs:

Constrained Local Models) 在非刚性物体的对齐和跟踪方面展示了更好的性能。对于非刚性物体的对齐和跟踪，一个主要的阻碍CLMs 进一步发展的问题是 :“ 如何根据局部搜索的响应，对全局形变的更新参数 (Global warp update) 进行联合优化？”之前的方法要么采用 general 的优化方式 ( 例如单纯形法 ), 要么采用基于图的优化技术。不幸的是，当应用于 CLMs 时，这些方法都存在问题。

本文提出了一种新的方法，强制每个局部 patch 的响应曲面为凸，这样就可以以一种高效的方式对全局形状更新进行优化。进一步，我们证明经典的基于 Lucas-Kanade 方法进行梯度下降的图像对齐可以看做本文提出的框架的一个特例。

最后，在非刚性的人脸对齐和跟踪方面，我们的方法在 Multi-PIE和 UNBC-McMaster 数据库上取得了更好的性能。

Overview of Constrained Local Models

(i) an exhaustive local search for feature locations to get the response maps

(ii) an optimization strategy to maximize the responses of the PDM constrained landmarks.

1{ ( | , )}ni ip l aligned I x

Saragih, J.M.; Lucey, S.; Cohn, J.F.; , “Face alignment through subspace constrained mean-shifts,” ICCV, 2009, pp.1034-1041

Key Steps of CLMs

Estimating patch/region experts Obtaining local responses Estimating PDM(point distribution

model) Constrained local model fitting



Key Steps of CLMs

Arbitrary classifier can be employed to learn patch experts within a CLM framework boosting schemes (e.g., AdaBoost, GentleBoost, etc.) relevance vector machine (RVMs)

A linear SVM classifier was chosen, due to computational advantages

Estimating patch experts



Key Steps of CLMs

Obtain local responses

(a) is the source image to be aligned, while the black box stand for the search window (25*25), the red cross illustrate the ground truth alignment.

(b) shows the local search responses using patch experts trained by 125

positive examples and 15k negative examples.

(b) shows the local search responses using patch experts trained by 125

positive examples and 8k negative examples.

(d) and (e) show the estimated logistic regression weight values of (b) and (c), respectively.

( )

1( 1| ( ))

1where is thematch-score

for thepatch-export

a f x bP y f x

ef( x)



Key Steps of CLMs

A point distribution model (PDM) is used for a parametric representation of the non-rigid shape variation in the CLM.

The non-rigid warp function can be described as , where , p is a parametric vector describing the non-rigid warp, andV is the matrix of concatenated eigenvectors. N is the number of patch-experts.

Principal component analysis (PCA) is then employed to obtain shape eigenvectors V that preserved 95% of the similarity normalized shape variation in the train set.

Estimating PDM



Key Steps of CLMs

Based on the patch experts, non-rigid alignment as be posed as the following optimization problem:

where is the inverted classifier score function obtained from applying the th patch expert to the source image patch intensity

The displacement is constrained to be consistent with the PDM

The matrix can be decomposed into submatrices for each patch expert, i.e.

CLM fitting

()kE

k ( )k k kY x x

x

V kV

1[ ,..., ]T T TNV V V

{ ( )}arg min k k kkp

E Y x V p

拟解决的问题 How to jointly optimize global warp update across all

local search responses?

In general, it is difficult to solve for p, as is a discrete function due to only taking on integral values and there is no guarantee for being convex.

()kE

()kEx

{ ( )}arg min k k kkp

E Y x V p

A Sub-optimal Approach Exhaustive Local Search (ELS)

Instead of optimizing for the holistic warp update p directly, ELS optimizes for N local translation updates by exhaustively searching local regions of the object

Where is the local warp update displacement of the kth region/patch (k=1,…N) within a local search region. Then

Where V is the matrix of concatenated eigenvectors. W is weighting matrix,

Y. Wang, S. Lucey, and J. Cohn., Non-rigid object alignment with a mismatch template based on exhaustive local search, In IEEE Workshop on Non-rigid Registration and Tracking through learning, 2007.

{ ( )}arg mink k kx

x E Y x x

kx

1( )p VWV VW z

1 1{ , , , , }

N Nx y x yw diag w w w w

本文的解决思路Learning from Lucas-

Kanade (1/3) Let us assume that we are attempting to solve for N

local translation updates as in the following equation

When a sum of squared differences (SSD) error function is employed:

where T is an arbitrary defined template. We no longer have to exhaustively search a local region around .

{ ( )}arg mink k kx

x E Y x x

2( ) ( )arg mink k k

x

x T x Y x x

kx


Kanade(2/3) Equation

can be rewritten by employing a first Taylor series approximation at .

which can be expressed generically in the form of a quadratic,

given,

2( ) ( )arg mink k k

x

x T x Y x x

( )kY x

2( ) ( )arg min T

k k kx

x D x G x x

2T Tk k kx A x b x c

2

2

where ( ) ( ) ( ),and ( )is the2 localgradient

( )matrix foreachsetof intensitycenteredaround .

k k k k

kk

k

D x T x Y x G x P

Y xP x

x

( ) ( ), ( ) ( ), ( ) ( )T Tk k k k k k k k kA G x G x b G x D x c D x D x


Kanade(3/3) Since is virtually always guaranteed of being positive

definite, this implies the quadratic is convex, and has a unique minima.

Since the summation of N convex functions is still a convex function, it is possible to solve not only for the local translation undates but the entire warp update explicitly,

where V is the matrix of concatenated eigenvectors describing the PDM, and the matrix A has the form

1 0

0 N

A

A

A

( )Tp VAV Vb

kA

p

1[ , , ]T T TNb b b

本文的解决思路Generic Convex Quadratic

Curve Fitting When is not a SSD classifier, but any function

that gives a low value for correct alignment,

For 2D image alignment, the problem can be further simplified as

where

()kE

2

, ,arg min ( ) 2

0

( ) { ( )}.

k k k

T TA b c k k k kx

k

k k k

E x x A x b x c

subject to A

whereE x E Y x x


Curve Fitting The above optimization is a quadratically constrained

quadratic program (QCQP) and in general costly to be solved directly.

So, is enforced to be a diagonal matrix with non-negative diagonal elements. More specially,

So,

kA

1111 22

22

0, , 0

0k

aA where a a

a

Convex quadratic fitting (CQF), which can be solved efficiently.


Curve Fitting Algorithm outline

进一步的改进 Robust error function

In particular, the robust error function can be defined as

Example FitsExamples of fitting local search responses: (a) is the local search responses in Figure 1(d) using patch experts trained by a linear support vector machine (SVM). (b-d) show the surface fitting results. More specifically, (b) picks the local displacement with the minimum response value in the search window, while (c) and (d) fit the local search response surface by a quadratic kernelin Equation 15 and a quadratic kernel with a robust error function in Equation 16, respectively. The brighter intensity means the smaller matching error between the template and the source image patch. In each search window, the red cross illustrates the ground truth location. As we can see, in most cases, the above three methods can all achieve good performance, while the proposed convex quadratic fitting (CQF) (c) and the robust convex quadratic fitting(RCQF) (d) methods are less sensitive to local minima than the exhaustive local search (ELS) method (b).

本文方法与已有方法的对比

本文可以借鉴的地方 Formulation

Demos

CLMs Demo: http://web.mac.com/jsaragih/iWeb/FaceTracker/FaceTracker.html

谢谢！

附录

ASM(1/7)

一个物体的几何描述分为两部分：相似变换（旋转、缩放、平移）形状

ASM(2/7)

ASM 的任务：得到姿态参数得到形状的低维表示，即参数 b

ASM 匹配的基本过程：

搜索在马氏距离下搜索与相应灰度梯度分布模型最匹配的特征点

调整对搜索得到的形状进行调整，以确保获得的形状是可用的

x u b

ASM(3/7)

ASM 模型：要完成 ASM 搜索与匹配的过程，必须要有相应的统计模型

做支撑 ASM 模型分为：

每个标注点的灰度梯度分布模型点分布模型

ASM(4/7)

每个标注点的灰度梯度模型的构建取标注点的 profile ，并计算得到该 profile 的归一化

的灰度梯度向量 Profile 灰度采样：梯度归一化

若训练集中有 M 个形状，每个形状有 N 个标注点，那么对于每个标注点，有协方差矩阵

，这 N 个协方差矩阵就构成了灰度梯度模型

,1i i N

1 2 Pr( , ..., )ofileLengthv v v

1 1

1( )( )

1

M Mj k

i i avgi i avgij k

VAR g g g gM

,1i i N

2 1 3 2 Pr Pr 1( , ..., )i ofileLength ofileLengthg v v v v v v '

Pr 1

1

| |i ofileLength

ijj

gg

g

ASM(5/7)• 点分布模型的构建

对于单个形状，每个标注点的坐标为将所有标注点的坐标串接起来，就得到一个形状向量

若训练集中有 M 个形状，那么我们就有 M 个形状向量

于是就可以训练形做 PCA ，得到能描述训练集形状变化的特征值与特征向量，即得到了点分布模型

( , ),1i ix y i N

1 1( , ,..., , )TN Nx y x y

1 1 1( , ,..., , )

MTN N j j

x y x y

ASM(6/7)

ASM 搜索对沿标注点的 profile 线的每个像素点取梯度向量搜索点为：

,1i i N j ijg1arg min( ) ( )ij avgi ij avgi

j

g g VAR g g

ASM(7/7)

ASM 调整假定经过一步搜索之后得到形状

将形状进行 PCA投影得到参数 b

利用 b重新计算形状

1 1( , ,..., , )TN NS x y x y

( )Tmeanb S S

new meanS S b

enforcing convexity for improved alignment with constrained local models authors: yang wang, simon...

Documents