3-d face recognition based on warped example faces

3-D Face Recognition Basedon Warped Example Faces

指導老師 : 萬書言老師報告學生 : 何炳杰報告日期 : 2010/05/05

1

Outline

• Abstract• Introduction• Generic Face Warping• Linear Combination of Face Warpings• Feature Extration Procedure• Feature Classification Using Mahalanobis Distance• Experimental Results• Conclusion

2

Abstract• In this paper, we describe a novel 3-D face recognition

scheme for 3-Dface recognition that can automatically identify faces from range images, and is insensitive to holes, facial expression, and hair.

• In our scheme, a number of carefully selected range images constitute a set of example faces, and another range image is chosen as a “generic face.”

• The generic face is then warped to match each of the example faces in the least mean square sense.

3

Abstract• Each such warp is specified by a vector of displacement

values.• In feature extraction operation, when a target face image

comes in, the generic face is warped to match it. The geometric transformation used in the warping is a linear combination of the example face warping vectors.

• Our technique is tested on a data set containing more than 600 range images.

• Experimental results in the access-control scenario show the effectiveness of the extracted features.

4

Introduction• FACE recognition has received broad attention in both

academia and industry due to its practical relevance to homeland security.

• Facial appearance can uniquely identify a person, and it is a primary factor that people use to recognize each other.

• Although human face images can be affected by illumination, facial expression, makeup, etc., they are noninvasive in nature and easy to be collected in environments where other biometrics (e.g., fingerprint, iris) require cooperation of the subjects.

5

Introduction• In this paper, we propose a scheme based on a number of

carefully selected 3-D faces called “example faces” and a 3-D face called a “generic face.” The set comprising the example faces is then defined as “example set.” In the design process, the generic face is warped to match each example face. Each such warping can be represented by a displacement vector.

• In a feature extraction operation, when a target face image comes in, the generic face is warped to match it.

6

Introduction• The geometric transformation used in the warping is

defined by a linear combination of those displacement vectors specified by the warpings from the generic face to the example faces.

• The coefficients in the linear combination are then used as features of the target image and passed to a classifier for recognition.

• To illustrate the idea of synthesizing 3-D faces based on a near combination of warps derived from example faces, consider the 3-D feature space shown in Fig. 1.

7

Introduction

8

Introduction

9

Here, we have three example faces, each of which corresponds to a particular geometric transformation (shown in Fig. 2), namely, the one that makes the generic face best match that example face.

Introduction• Each axis represents one coefficient in a three-term

weighted sum. Each example face lies at 1.0 along its corresponding axis.

10

Further, any point in that 3-D space corresponds to a particular linearcombination of those three warps (shown in Fig. 3)

Introduction• In feature extraction operation, the algorithm adjusts the

coefficients to minimize the root-mean-square (rms) error between the target face and the warped generic face.

• The coefficient vector corresponding to the minimum then constitutes a set of feature values that represents that face.

11

Introduction

12

The feature extraction procedure of our scheme is illustratedin Fig. 4.

Introduction• This procedure starts with an initial coefficient vector , typically all equal to 1/K .

• Based on this initial coefficient vector, the generic face is warped to generate a synthesized face . Then, the rms error between and the target face T is computed.

13

1S

1S

Introduction• To summarize, our proposed scheme can be outlined as follows. 1) During the design process, warp a generic model of a face to

match each K example face, retaining the parameters of each K such warps in a warp vector.

2) In a feature extraction procedure, warp the generic face using a geometric transformation that is a linear combination of the warp vectors that were developed for the example faces.

3) Adjust the coefficients in the weighted sum to minimize the rms error between the input target face image and the warped generic face.

4) Take the coefficients that result in minimum rms error as extracted features that describe the face.

5) Use a linear classifier based on Mahalanobis distance to classify the extracted features.

14

Generic Face Warping• Preface :- The goal of generic face warping is to establish point-

to-point correspondences between the generic face and example faces.

- we can consider the warped generic face based on a combination of face warps derived from different example faces as a synthesized face.

- The main thrust is the generic face warping procedure that establishes the point-to-point correspondence between the generic face and example faces.

15

Generic Face Warping• A. Necessity for Alignment:- we need to first develop the proper face warpings

between the generic face and example faces.- In other words, we need to find out the point-to-point

correspondence between the generic face and the example faces. Otherwise, when different warps derived from example faces are combined together, points with different physical meanings will be added up, resulting in blurred faces.

16

Generic Face Warping

17

For instance, if we develop two face warpings between thegeneric face and two example faces [shown as example faces in Fig. 5(a) and (b)] without alignments, a combination of the face warpings will generate a blurred face shown in Fig. 5(c).


18

With the alignment, however, pixels with the same physical meaning are aligned together and generate a valid human face, as shown in Fig. 5(d).


19

Generic Face Warping• B. Landmark Points:

20

In order to find out the point-to-point correspondences between the generic face and example faces, we first manually define 70 landmark points (e.g., eye corners and mouth corners) on each K example face and the generic face, as shown in Fig. 6.

Note that the number of landmark points can vary.

Generic Face Warping• B. Landmark Points:• In our face recognition scheme, we find 70 landmark

points are enough to accurately describe the correspondences between the generic face and an example face.

• The and coordinates of these 70 landmark points are concatenated to generate position vectors and

, i = 1,…, K

21

x y( )LX i

( )LY i


22

(x1, y1)

(x4, y4)

(x2, y2)

(x3, y3)

Positionvector Position

vector

x

y

z

Generic Face Warping• B. Landmark Points:• The position vectors of the landmark points on are

denoted as and and .• : The generic face. ( 共通臉 )

23

gX gYG

G

With these position vectors, the displacement vectors and ( i = 1, …,K )are defined as

( )LX i ( )LY i


: the weighting coefficients意義 :

24

Based on these displacement vectors, we canfurther generate new displacement vectors

iw

Depending on , and actually represent a set of displacements of the 70 landmarkpoints on the generic face .

1 2( , ,..., )tKW w w w ( )lX W ( )lY W

G

Generic Face Warping• C. Face Warping:- In this section, we describe the warping procedure that calculates the displacements of every point on the generic face based on the displacement vectors , .

25

G ( )lX W

( )lY W

a point in the generic face will be mapped to a point in the warped face .

( , )x y G( ( , , ), ( , , ))x x W x y y y W x y

( )S W

after this procedure

Generic Face Warping• C. Face Warping:- Image size:750 * 500 range image size.- Process:We first divide the generic face into Delaunay triangles based on the 70 landmark points defined by and , as shown in Fig. 7.

26

gX gY

Generic Face Warping• C. Face Warping:

27Fig. 7. Range face image divided by Delaunay triangles.

Generic Face Warping• C. Face Warping:- The coordinates and of the triangle vertices are

then mapped to and - We can focus on a pair of such corresponding

Delaunay triangles as shown in Fig. 8, in which the triangle are mapped to the triangle .

•

28

gXgY

( ) ( )l g lX W X X W ( ) ( )l g lY W Y Y W

1T 2T

Generic Face Warping• C. Face Warping:- Now given a point inside , we want to find out the corresponding position in .- If we restrict ourselves to linear mappings, we have

: the coefficients that need to be determined.

29

( , )x y 1T( ( ), ( ))l lx W y W 2T

( ) ( )a W f w

Generic Face Warping• C. Face Warping:- Because of the correspondences between vertices of and ,we have

30

1T 2T

Generic Face Warping• C. Face Warping:- From (4) and (5), we can obtain

31

Generic Face Warping• C. Face Warping:- Note that is invertible as long as has a nonzero

area, so we have

32

M 1T

Generic Face Warping• C. Face Warping:- Let • • • , from (8) and (9), we can rewrite (3)

as

33

1 2 3( ) ( ( ), ( ), ( ))l l l l ttX W x W x W x W

1 2 3( ) ( ( ), ( ), ( ))l l l l ttY W y W y W y W

P: 位置向量 since P= then P=( , ,1)tx y1 2 3

1 2 3

1 1 1

x x xy y y

1 1

2 2

3 3

111

t

x yP x y

x y

P ( , ,1)tx y

Generic Face Warping• C. Face Warping:- If we draw vertical and horizontal lines passing through

the landmark points of the generic face , we can further divide into more than 300 small grids, as shown in Fig. 9.

34

GG

Generic Face Warping• C. Face Warping:- Suppose a particular grid has four vertices , i = 1, 2, 3, 4- Let , i = 1, 2, 3, 4 be the corresponding

displacements of the vertices, then the displacements of a pixel with the position inside a grid.

35

( , )i ix y

( ( ), ( ))i ix W y W

( , )x y

( , )x y

1 1( ( ), ( ))x W y W 2 2( ( ), ( ))x W y W

3 3( ( ), ( ))x W y W 4 4( ( ), ( ))x W y W

G


36

C. Face Warping:

( , )x y

1 1( ( ), ( ))x W y W 2 2( ( ), ( ))x W y W

3 3( ( ), ( ))x W y W 4 4( ( ), ( ))x W y W

( ( , , ), ( , , ))x W x y y w x y

W: the weighting coefficients

Linear Combination of Face Warpings

• A. Synthesized Faces From a Linear Combination of Face Warpings:

- The displacement images and of example faces.

- : The displacement image of the ith example face .

- x and y : The coordinates of the image.

37

( , )iX x y ( , )iY x y ( 1,..., )i K

( 1,..., )iE i K

iE



- From (12), we can reconstruct the example face as follows:

38

( 1,..., )iE i K



- The obtained displacement images of the example images , and

to generate new displacement images

39

( , )iX x y ( , )iY x y ( , )iZ x y ( 1,..., )i K



- Where , , and can additionally be used to create a new synthesized face

:

40

( , , )X W x y ( , , )Y W x y ( , , )Z W x y

( )S W


41

If we allocate each displacement image as the same weight 1/K, we can generate the average warped generic face, as shown inFig. 10.

Linear Combination of Face Warpings• B. Selection of Example Faces:

42

2-D feature space

cloud

think as “points”


• B. Selection of Example Faces:- The larger the “cloud,” the wider range of valid

synthesized faces will be generated.- Hence, we should select the extreme faces (e.g., long

faces and short faces) as example faces, so that the “cloud” composed by the example faces will be as large as possible

43

言外之意，該系統的人臉資料庫中，長臉與短臉數量較多。

弦外之音


• From Section III-A, we get a range face E,

• And

• V : Let the length of vector V be L.

44

XYZ

We can odtain the corresponding displacement images by these.

XYZ

can be concatenated as a vector V


• B. Selection of Example Faces:- By calculating the mean vector of these H vectors , we can obtain the data matrix :

- H : range images in the data set- The covariance matrix of D can therefore be expressed

as .

45

VV ( 1,..., )i i H

1(V V,...,V V)HD

(1/ ) TP H DD

Note that the covariance matrix P is L*L, where L can be a very large number.


• B. Selection of Example Faces:- In order to save calculation complexity, we compute

the eigenvectors from .- Let the corresponding eigenvalue of an eigenvector be . We have

: eigenvector( 特徵向量 ) : 該特徵向量的特徵值 : matrix( 矩陣 )

46

U ( 1,..., )li i H (1/ )l TP H DD

Uli i

Uli

iD


• B. Selection of Example Faces:- By multiplying the matrix D to the left side of the above

equation, the following equation can be obtained:

- where is clearly an eigenvector of the covariance matrix P.

47

UliD

We can then sort the eigenvectors according to the values of their corresponding eigenvalues, from large to small.


• B. Selection of Example Faces:- In order to reduce the noise in target images caused by

facial expressions, we add example faces with facial expressions to the example set, as illustrated in Fig. 12.

48without experssions with experssions


• B. Selection of Example Faces:

49

with neutral expressions

with laughing expressions

Feature Extration Procedure

• The purpose : - 給予一個新的目標人臉影像 T ，特徵擷取過程的目標是在擷取一個可信賴的人臉特徵點，而這些被擷取下來的特徵點可以足夠與其他人臉做區別。- 基於從例子人臉校準後的臉部變型，作者設計出一個可以使已變型的共通人臉 ( 合成臉 ) 盡可能的與目標人臉相似的一個演算法。- 在這個小節裡，作者首先引入 cost function 的概念來描述合成後的人臉與目標人臉的差別。

50


• A. Cost Function : - Suppose we have K example face :- : the generic face – 原始人臉影像- T : a target face – 欲變形成的目標臉- W : a weighting vector – 權重向量- S(W) : the synthesized range face – 由原始臉合成的人臉

51

( 1,..., )iE i KG


• A. Cost Function :

52

For a certain position on , the squared range difference between S(W) and T becomes that.

( , )x y G


• A. Cost Function : - In order to measure the similarity between S(W) and T, we define the cost function as the sum of the squared

difference :

- where is a set containing selected points on the generic face.

(x, y)

53

Feature Extration Procedure• A. Cost Function :- 圖解 :

• 集合

54

a set containing selected pointson the generic face

generic face

Feature Extration Procedure• B. Matching Region Selection : - Dynamically select points in so that holes will be

avoided in the matching procedure.- To understand the adaptive hole avoiding strategy, let

us consider picking a point P = (x, y) the point set of the generic face, as illustrated in Fig. 14. 集合 P(x, y)

55


• B. Matching Region Selection : • 說明 : 於集合中選取點 P (x,y) 一個點 P (x,y), 其中

• 集合中的點為共通臉 (G)• 上的點。根據目前求得的的W 值，點 P 將會與目標

• 臉上的點• 相互比較。

56P(x, y) P ( ( , , ), ( , , ))l x X W x y y Y W x y


• B. Matching Region Selection : - In order to avoid the facial hair and facial expression

in a target face, we define the matching region (shown in Fig. 15)

57


• C. Optimization :- we choose Newton’s method , which is based on the

calculation of derivatives of .- The derivatives of with respect to the ith

weighting coefficient are given as

58

( , )C W

( , )C W iw


• C. Optimization :

59


• C. Optimization :- Update the next weighting coefficients as follows:

- is the inverse Hessian matrix and is the learning rate.

- : learning rate - 學習率 , 在類神經網路的訓練過程中是一個非常重要的參數。學習率影響著類神經網路收斂的速度，若學習率選擇較大則類神經網路收斂的速度將變得較快，反之，較小的學習率會使得類神經網路的收斂速度變慢。選擇太大或太小的學習率對類神經網路的訓練都有不良的影響。

Feature Classification Using Mahalanobis Distance

• A. Feature Preprocessing :- Let us take a look at the examples on the plane as

illustrated in Fig. 16. - As shown in Fig. 16, these two vectors have the

following two properties:(1) The covariance of samples is maximized by projecting thesamples to the normal vector .(2) The normal vector is orthogonal to vector .

61

XY

Yl

Xl

Xl


• A. Feature Preprocessing :

62

X’ 與 Y’ 對於 XY 平面而言均為法向量，且 Y’ 與X’ 垂直

XY 平面 ( 紅色虛線處 )

Sample face : example face


• B. Mahalanobis Distance ( 馬氏距離 ) :

- 參數 : : Particular human face. : The mean vector. x : Feature vector. : Covariance matrix. ( 協方差矩陣 )- 原始馬氏距離公式 :

63

P


• B. Mahalanobis Distance ( 馬氏距離 ) : - 依照 paper 的說法， Fig.17 的馬氏距離圖可以用公式 (23) 來表達 :

64

Feature Classification Using Mahalanobis Distance • B. Mahalanobis Distance ( 馬氏距離 ) :

- 引入馬氏距離的目的 : 利用馬氏距離，將一系列線性產生的人臉影像資料作為分類的依據。- 於 paper 中作者提出兩個人臉影像分類 : : Represents the class of a particular human face . - : Represents all other sample range images.- Given a threshold h, we can assign a sample x to if

its Mahalanobis distance ; otherwise, x will be assigned to .

65

P

Experimental Results • A. Data Acquisition : - All range images are obtained from a 3-D camera

system.- There are a total of 650 range images in our data set

containing 113 different human objects.- About 100 images contain faces with facial hair,

expressions, and holes.- Each range image is 750*500 in size.

66

Experimental Results • A. Data Acquisition :

67

Experimental Results • B. Accuracy and Robustness of the Optimization : - In the experiments : (i) First measure the accuracy between the positions of

landmark points in the synthesized face and target face.

( 一開始，先測量出合成臉與目標臉上的特徵點位置。 )(ii) Then, the distribution of range differences between

the synthesized face and target faces are given. ( 接著，合成臉與目標臉上的特徵點位置差異的分佈，即可得知。 )

68

Experimental Results • B. Accuracy and Robustness of the Optimization : - In the experiments : (iii) Use the weighting coefficients to calculate the

location of the 70 landmark points on the synthesized face. ( 使用權重係數去計算出位於合成人臉影像上 70個特徵點的位置。 )(iv) Compare the location of the landmark points on the

synthesized face with the real location of the landmark points on the target face .

( 最後，比較合成臉上特徵點位置與目標人臉上實際的特徵點位置。 ) 69

*w

Experimental Results • B. Accuracy and Robustness of the Optimization :

70

The extracted landmark points are shown on texture images and connected with lines.

Experimental Results • B. Accuracy and Robustness of the Optimization : - The accuracy of the matching procedure can then be

evaluated by . - the smaller the average range difference is, the higher accuracy the matching procedure achieves.

- : See paper Fig. 15, please. (p.521)

71

Experimental Results • B. Accuracy and Robustness of the Optimization : - Based on our database that contains more than 600

range images, we test the matching procedure and give the distribution of the average range difference in Fig. 20.

72

Experimental Results • B. Accuracy and Robustness of the Optimization : - It is seen that after the matching procedure, about 95%

resulted average range differences are below the 8 range unit, and less than 1% are larger than the 9 range unit.

73

Experimental Results • C. Feature Extraction Schemes for Comparison : - The first chosen feature set is the face profiles, which

can be generated directly from the target image .

74

T

Experimental Results • C. Feature Extraction Schemes for Comparison : 1. The tip of the nose of T will be fixed to the center of the images. 2. We can obtain the vertical and

horizontal profiles of horizontal profiles of T directly following the vertical and

horizontal horizontal line divide the image across its center.

75

Experimental Results • C. Feature Extraction Schemes for Comparison : - The reason for choosing face profiles?Ans : Because they contain some important detailed

information that differentiates one person from another. In addition, profiles are relatively insensitive to

facial expressions, holes, and hair.

76

Experimental Results • D. Classification Performance : - The settings of the experiments are shown in Fig. 22.

77

Experimental Results • D. Classification Performance : - The samples : a. The authorized users : 8 human subjects.b. The imposters users : 115 human subjects. c. Each authorized user has 40 range images.d. Each imposter only has a few images in the database. e .Total: 330 range images from imposters.

78

Experimental Results • D. Classification Performance : - The experiment process : (i) First, range images in the database are processed by

the three different feature extraction schemes. ( 首先，資料庫中的範圍臉 -range images 影像用三種不同的特徵擷取系統進行特徵點的處理。 )(ii) After the feature extraction procedure, three different

feature sets are obtained. ( 特徵擷取處理完後，三個特徵系統的擷取結果便可取得。 )

79

Experimental Results • D. Classification Performance : - The experiment process : In the training step : - 基於 training set 樣本中，對於每一位予以授權的八位使用者，作者先計算出其人臉影像的協方差矩陣與平均值。而這些求出來的值，將來會被用來作為馬氏距離的分類依據。During the testing step :- 接著，在測試資料階段：一組輸入進來的樣本影像與八張授權過的人臉影像經由馬氏距離的計算後，由馬氏距離的門檻值判斷其這張測試的樣本影像分類是屬於授權的人臉影像或是外來者的人臉影像。

80

Experimental Results • D. Classification Performance : - The surface matching method : In the matching procedure, given a testing image, it

will be matched to all range images in the training set. The matching distance is then used as a metric to make a classification decision.

( 在比對程序裡，給予一張測試的人臉影像，這張測試的影像將會去一一比對所有的 training set 影像資料。而比對過程中所產生的點距離，將來則用來當成矩陣形式使用，作為影像分類決策的依據。 )

81

Experimental Results • D. Classification Performance : - ROC - (a) : Based on the weighting coefficients In (a) : the hit rate : 97.2% (at the false alarm rate 2%)

the hit rate : 98.5% (at the false alarm rate 5%)

82

Experimental Results • D. Classification Performance : - ROC - (b) : Based on the profiles. In (b) : the hit rate : 93.4% (at the false alarm rate 2%)


83

Experimental Results • D. Classification Performance : - ROC - (c) : Based on the eigenface method. In (c) : the hit rate : 91.2% (at the false alarm rate 2%)


84

Experimental Results • D. Classification Performance : - ROC - (d) : Based on the surface matching method. In (d) : the hit rate : 75.1% (at the false alarm rate 2%)

the hit rate : 89% (at the false alarm rate 5%)

85

Total ROC (a) ~ (d)

86

Experimental Results • D. Classification Performance : EER 說明 : 錯誤拒絕率（ False Rejection Rate, FRR） : 應該辨識通過的卻沒通過。錯誤接受率（ False Acceptance Rate, FAR） : 應該不能通過辨識的卻通過了。等錯誤率（ Equal Error Rate, EER） : FAR及 FRR構成的曲線會交會於一點，也就是兩種辨識錯誤率相同的點， ERR 的值越低表示辨識系統的效能越好。

87

Experimental Results • D. Classification Performance :

88

misclassified faces

corresponding synthesizedfaces

mistakenly classified

3-d face recognition based on warped example faces

Documents