facial point detection using boosted regression and graph models

Facial Point Detection using Boosted Regression and Graph Models

Authors: Michel Valstar,Brais Martinez, Xavier Binefa, Maja Pantic讲解人 : 赵小伟

提纲作者信息文章信息背景知识拟解决的问题与采用的思路实现细节实验结论

第一作者 Michel Valstar

Research associate in Maja Pantic's HCI^2 lab at the Computing Department of Imperial College London, UK

Research Interest Automatically recognize facial expressions from face

video

Publication CVPR’06, CVPR’10

Homepage http://www.doc.ic.ac.uk/~ mvalstar/index.html

第二作者 Brais Martinez

PhD Student, Universitat Pompeu Fabra

Research Interest Object Tracking Facial Feature Detection and Tracking Thermal Imagery

Publication 2 CVPR’10 , PR’08, ICIP’06

Homepage http://cmtech.upf.edu/?page_id=90

第三作者 Xavier Binefa Valls

Associate Professor, Information Technology and Telecommunication Department of the Universitat Pompeu Fabra

Research Interest Motion Detection and tracking, Machine Learning Face and Gesture recognition, Digital Libraries Human computer interaction, Sensor Fusion

Homepage http://cmtech.upf.edu/?page_id=84

第四作者 Maja Pantic

Imperial College London: Reader in Multimodal Human-Computer Interaction

University of Twente: Professor in Affective Behavioural Computing

Research Interest Face and body gesture recognition, Human-computer interaction (HCI), Affective computing, Educational software, E-

learning tools, Intelligent systems, Machine learning

HomePage http://www.doc.ic.ac.uk/~maja/

文章信息文章出处

CVPR 2010

相关文献 [23] D. Vukadinovic and M. Pantic, “Fully

automatic facial feature point detection using gabor feature based boosted classifiers,” In Proc. Systems, Man and Cybernetics, vol. 2, pp. 1692–1698, 2005.

Abstract Finding fiducial facial points in any frame of a video

showing rich naturalistic facial behavior is an unsolved problem. Yet this is a crucial step for geometric-feature-based facial expression analysis, and methods that use appearance-based features extracted at fiducial facial point locations.

In this paper we present a method based on a combination of Support Vector Regression and Markov Random Fields to drastically reduce the time needed to search for a point’s location and increase the accuracy and robustness of the algorithm.

Using Markov Random Fields allows us to constrain the search space by exploiting the constellations that facial points can form.

Abstract The regressors on the other hand learn a mapping

between the appearance of the area surrounding a point and the positions of these points, which makes detection of the points very fast and can make the algorithm robust to variations of appearance due to facial expression and moderate changes in head pose.

The proposed point detection algorithm was tested on 1855 images, the results of which showed we outperform current state of the art point detectors.

摘要在具有丰富的自然面部行为的视频帧中进行面部关键特征点的定位是

一个尚未解决的问题。然而，对基于几何特征的面部表情分析以及需要从面部关键特征点提取表观特征的方法而言，面部关键特征点的定位是一个很重要的步骤。

本文提出了一种结合 SVR 和 MRF 的面部关键特征点定位方法。该方法大大降低了搜索特征点的时间，并且提高了算法的精度和鲁棒性。

一方面，使用 MRF 对面部关键特征点的分布进行建模，以此来限制特征点的搜索范围。

另一方面，通过 SVR 学习到了特征点周围区域的表观信息与特征点位置的映射关系。该方法可以更快的检测特征点，并且对由面部表情和头部姿态的适度变化引起的表观变化比较鲁棒。

我们在 1855 幅图像上测试了提出的面部特征点检测算法，实验表明，本文的算法超越了当前 state-of-the-art 的算法。

AdaBoost-based Facial Landmark Localization

Multi-Scales Detection

Face Detection & Normalization

Candidate Points Search Region Determination

Candidate Points Fusion

Real AdaBoost Classisiers

......

......

Negative Samples

Positive Samples

Real AdaBoost Learning

Preparing Samples

Feature Extraction

拟解决的问题 None but [23] is able to detect all 20 facial points

necessary for automatic expression recognition No previous work has reported to be able to robustly

handle large occlusions such as glasses, beards, and hair that covers part of the eyebrows and eyes

None have reported to detect facial points robustly in the presentence of facial expressions

22 fiducial facial feature points (including pupils)

本文的主要思想 Iteratively using Support Vector Regression and local

appearance based features to provide an initial predictions of 22 points

Then, the Markov Network is applied to ensure the new locations predicted by SVR regressors form correct point constellations

SVR regression MRF points modelThe output of the SVRs to detect an pupil

文章结构 Introduction BoRMaN point detection

A priori probability Regression prediction Spatial relations Point detection algorithm Local appearance based features and AdaBoost feature

selection

Experiments Conclusions and future work

实现细节 A priori probability Regression prediction Local appearance based features and

AdaBoost feature selection Spatial Relations Point detection algorithm

Regression prediction The localization problem is formulated as finding the

vector that relates a patch location to the target point .

This problem is decomposed into two separate regression problem Regressor is tasked with finding the angle of Regressor is tasked with finding the length of

As we can see, the regressors give a good yet not a perfect indication of where the target point is. Note that although the location of the pupil is a global

minimum, the predicted distance at that location is not zero.

Lv

T

R

R

v v

Regression prediction The error of the estimates

Impression of the regressors output Great errors which are not merely impressions

The output of the SVRs to detect an pupil

Such errors can be removed by using a iterative procedure.

Spatial restrictions on the location of each facial point depending on the other facial points are applied to solve this problem.

Local appearance based features and AdaBoost

feature selection Haar-like filters are adopted as the descriptors of

local appearance The reason for this is

Show that the success of the proposed approach is due to the idea of tuning the point detection problem from a classification procedure into a regression procedure, and not due to asome highly descriptive appearance feature

Exploring the integral image The regression performance decrease when the

dimensionality of the training set is too large AdaBoost is used to select features

Spatial Relations Each relative position of a pair of points is a

vector pointing from one facial point to another The relation between two vectors and is

described by two parameters The relation between their angles The relation between their lengths

{ , }i j ,i jr

, ,i j k lR ,

,

i j

k lR

,i jr ,k lr

Relation between two vectors

(0,0)

Spatial Relations Variables such as and are modeled as a

Sigmoid function. If a variable takes its value in , then

R R

Illustration of Sigmoid function, cited from Wiki

( ) (min( , )), ( ) ( ) 0.5sigmS x P x m x m whereS m S m

[ , ]m m

Spatial Relations Once the pairwise relations are defined, the joint

probability of a configuration is modeled by a Markov Random Field.

The nodes correspond to each of the relative positions

Relation between and is modeled as

,i jr

, , ,( , )i j i j i jr , , ,( , )k l k l k lr

, , , ,( , ) ( , )ang i j k l dist i j k lS S

Point detection algorithm

Flow of algorithm

本文方法与已有方法的对比

Distance Metric:

实验结果

本文可以借鉴的地方 Regression instead of classification Markov Random Field to model the

constellation of facial points Select features by AdaBoost

谢谢！

附录

Introduction of AdaBoost(1/6)

AdaBoost AdaBoost 通过对一些弱分类器 (weak classifier) 的

组合来形成一个强分类器 (strong classifier), “ 提升(boost)” 弱分类器得到一个分类性能好的强分类器

每一个弱分类器都对前一个分类器错误分类的样本给与更多的重视


AdaBoost 弱分类器其中， h表示弱分类器的响应值， θ为正例反例判

别阈值， f表示特征响应值

PositiveNegative

otherwise ,0

(x) if ,1)( jj

j

fxh


AdaBoost 训练过程输入

样本集合 (x1,y1), (x2,y2), ..., (xn,yn)

训练参数：样本权值 wi、分类器层数 T等等输出

一个由很多弱分类器线性组合得到的强分类器


分类错误率是否达到？

选择错误率最小的弱分类器更新强分类器

样本权值更新，分类正确的样本权值减小

遍历所有特征，分别计算以每个特征作为弱分类器的分类错误率

否

输出强分类器是

Introduction of AdaBoost(5/6) AdaBoost 训练过程

For t=1,...,T 1. 归一化权重，使得 wt为一个概率分布：

2. 对每个特征 j, 训练一个弱分类器 hj, 计算其带权重的错误率

3. 选择误差最小的弱分类器 ht加入强分类器 4. 更新每个样本的权重

,,

,1

t it i n

t jj

ww

w

, 1

| ( ) |n

j t i j i ii

w h x y

1-, , ,

1ie t

t i t i t tt

w w


AdaBoost 强分类器

1 1

1 if log ( ) 0.5 log( )

0

T T

t t tt t

h xH x

Otherwise

Haar-like Feature(1/2)

Haar-like feature 白色矩形像素和减去黑色矩形像素和

Haar-like Feature(2/2)

Haar-like feature 计算矩形内部像素灰度值的和定义积分图

计算 D内部像素灰度和 4 + 1 - 2 - 3

,

( , ) ( , )x x y y

ii x y i x y

facial point detection using boosted regression and graph models

Documents