1

A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions

Zhihong Zeng, Maja Pantic, Glenn I. Roisman, Thomas S. Huang

Reported by Chengsheng Mao

2011 年 1 月 11日

2

The Description of Emotion

• Discrete categories description: the most popular example of this description this description is the basic emotion categories, which include happiness, sadness, fear, anger, disgust, and surprise. This description of basic emotions was specially supported by the cross-cultural studies conducted by Ekman [40], [42].

• Dimensional description: the evaluation and activation dimensions are expected to reflect the main aspects of emotion. the evaluation and activation dimensions are expected to reflect the main aspects of emotion.

3

Audio and/or Visual Databases of Human Affective Behavior

4

Related work

5

Challenges

• Data collection for emotion recognition:– Spontaneous versus posed– Lab setting versus real-world– Expression versus feeling– Open recording versus hidden recording– Emotion-purpose versus other-purpose

• Labeling date for emotion recognition：– If constantly asks a user for his/her emotion, we can be quite sure that

eventually the response would be that of anger or annoyance.– Further research is required to achieve maximum utilization of

unlabeled data for the problem of emotion recognition

6

The research method on emotion

Experiment design

Signal preprocessing

Feature extraction

Original

signalsP

ure signals

1. Emotional model2. SAM (self assessment Manikin)3. Others

Data mining

Features

1. Referencing2. De-noise

1. linear (AR model, FFT, ...)2. non-linear (complexity, entropy, ...)

1. data preprocessing2. classification and prediction3. cluster analysis4. discriminant analysis

7

Data mining

• Data preprocessing– Data cleaning is applied to remove noise and correct

inconsistencies in the data.– Data transformations, normalization may improve the

accuracy and efficiency of mining algorithms.– Data reduction techniques can be applied to obtain a

reduced representation of the data set that is much smaller in volume, yet closely maintains the integrity of the original data.

8

Classification and prediction

• Classification and prediction– A classifier or predictor based on a certain algorithm is built

by analyzing or learning from a training set made up of database tuples and their associated class labels or values. (supervised learning)

– For classification, the classifier is used to classify the test data. Then the classification accuracy is calculated to estimate the classifier.

– For prediction, the values are predicted through the predictor and then an error based on the difference between the predicted value and the actual known value is computed to estimate the predictor.

9

Cluster analysis and discriminant analysis

• Clustering is the process of grouping the data into classes or clusters, so that objects within a cluster have high similarity in comparison to one another but are very dissimilar to objects in other clusters. (unsupervised learning)

• Discriminant analysis find the discriminant functions based on the training datum and their class labels. Then classify the data unknown category according to the discriminant functions.