Hierarchical Matching with Side Information for Image Classification

Authors Qiang Chen et al.

Presenter Hyung-seok Lee (이형석 )

CVPR 2012

2

Motivation (1/2)• Bag-of-Words (BoW)• Frequency histogram of visual words• Not emphasize any elements with regard to image layout

3

Motivation (2/2)• Spatial Pyramid Matching (SPM)• Partitioning the image plane into fine sub-regions• Not optimum for object-centered recognition problem

Beyond Bags of Features : Spatial Pyramid Matching for Recognizing Natural Scene Categories, Svetlana Lazebnik et al.

4

Main Idea• Generalized Hierarchical Matching (GHM)• Extends the popular matching work• Integrate prior knowledge for enhancing feature matching

side information

5

Overview

Im-age

Hierarchi-cal cluster-

ing

Hierarchical structure

representation

Matching over each cluster

Local features &

side informa-tion

6

Side Information (1/4)• Object Confidence Map• Object position should be extremely beneficial• Shape model / Appearance model

7

Side Information (2/4)• Shape-based object detection : HOG features

• Appearance-based object detection : BoW features

Object Confidence Map

8

Side Information (3/4)• Visual Saliency Map• Aim to detect the important content in images• Explain human visual search strategies

SUN : A Bayesian Framework for Saliency Using Natural Statistics, Lingyun Zhang et el.

9

Side Information (4/4)• Use the saliency model SUN (Saliency Using Natural statistics)• Based on natural image statistics

10

Side Information Combination• Combine object confidence map & visual saliency map• Design a 3 level hierarchical structure

- Object confidence map is used in level 2

- Background area will be further utilized in level 3

11

Encoding & Matching (1/2)

12

Encoding & Matching (2/2)• Encoding is operated on each cluster• Matching over each corresponding cluster

13

Experiments (1/2)• Caltech-UCSD Birds 200 & Oxford Flowers 17 and 102

14

Experiments (2/2)• VOC 2007

• VOC 2010

15

Conclusion• Propose the Generalized Hierarchical Matching framework

- Extends the popular pyramid matching

• Two novel kinds of side information are introduced to enhance object-oriented image classification tasks

- Object confidence map- Visual saliency amp