what, where & how many? combining object detectors and crfs

2023年4月21日 星期五

1

What, Where & How Many?Combining Object Detectors and CRFs

L’ubor Ladický, Paul Sturgess, Karteek Alahari, Chris Russell, and Philip H.S. Torr

Lecturer ： Zhiguo Ma

Outline

• Authors

• Abstract

• Background

• Hierarchical CRF

• Object detector potential in CRF

• Experiments & Conclusion

2023年4月21日 星期五

2

作者介绍

L’ubor Ladický8 papers on CVPR,ICCV,BMVC,ECCV ,etc.

Best paper of BMVC 2010 & ECCV 2010

Website ： http://sots.brookes.ac.uk/lubor/

No information for Paul Sturgess & Chris Russell

2023年4月21日 星期五

3

Karteek Alahari

2023年4月21日 星期五

4

10+ papers on ACCV, ICPR, CVPR, BMVC,PAMI, ECCV, etc.Website: http://www.di.ens.fr/~alahari/

Philip H.S. Torr

PhD at the Robotics Research Group of the University of Oxford.

Oxford as a research fellow, and is currently a Visiting Fellow in Engineering Science at the University of Oxford

Research scientist for Microsoft Research

Many papers on Journal & conference in fields of CV,ML, PR.

2023年4月21日 星期五

5

Abstract

Computer vision algorithms for individual tasks such as object recognition, detection and segmentation have shown impressive results in the recent past. The next challenge is to integrate all these algorithms and address the problem of scene understanding. This paper is a step towards this goal. We present a probabilistic framework for reasoning about regions, objects, and their attributes such as object class, location, and spatial extent. Our model is a Conditional Random Field defined on pixels, segments and objects. We define a global energy function for the model, which combines results from sliding window detectors, and low-level pixel-based unary and pair wise relations. One of our primary contributions is to show that this energy function can be solved efficiently. Experimental results show that our model achieves significant improvement over the baseline methods on CamVid and PASCAL VOC datasets.

2023年4月21日 星期五

6

摘要针对单独任务（如物体识别、检测和分割）的计算机视觉算法，在近几年取得很大的进步。下一个挑战是整合这些算法，解决场景理解的问题，本篇文章是向此目标前进的一步。我们提出了一种概率性框架用于推断区域、物体及其属性（如物体类别，位置及空间范围等）。我们的模型是一个定义在像素、区域、物体上的条件随机场。模型定义了一个全局能量函数，整合来自滑动窗口物体检测器、底层像素级的一元和二元信息。我们的一个主要贡献是展示这个能量函数可以被有效地求解。在 CamVid 及 PASCAL VOC 数据集上的结果显示，我们的模型比基准算法获得了很大的性能提升。

2023年4月21日 星期五

7

Background

2023年4月21日 星期五

8

（ a ）原始图像 （ b ）物体类别分割 （ c ）物体检测 （ d ）检测与分割结合（本文）

物体类别分割会丢失一些物体，且不提供物体数目信息；物体检测能检测到此类物体，但不提供前、背景分割结果。整合分割与检测，可以解决上述问题。

Related Work

Stuff and ThingsStuff: homogeneous or reoccurring pattern of fine-scale properties, but no specific spatial extent or shape

Things: have distinct size and shape.

Object class segmentationSuccessful on stuff, but fails on things

Foreground (thing) object detectionGood at things, but fail on stuff, which is amorphous

2023年4月21日 星期五

9

CRF

Label set object class( such as car, airplane, bicycle, etc.)

Random variables Image pixel

Clique c set of pixels conditionally dependent on each other

Labeling x any possible assignment of labels to pixels2023年4月21日 星期五

10

1 2{ , ,..., }nL l l l

1 2{ , ,..., }nX X X X

Posterior distribution & energy of CRF

2023年4月21日 星期五

11

Labeling

Data

Normalized factor

Clique Set Potential function

*

Engery Function: ( ) log Pr( | ) log ( )

MAP estimation of X: argmax Pr( | ) arg min ( )

c c

x L x L

E x X D Z xc C

X x D E x

1Pr( | ) exp( ( ))c c

c C

X D xZ

Potentials in energy function

Unary potentialLocal feature responses , the likelihood of a pixel taking a certain label

Pairwise potentialEncourage neighboring pixels take the same label

Higher order potential between segmentsModel relationship between segments, object, etc.

Color potential for instance of objectsForeground and background estimation

2023年4月21日 星期五

12

Object detector potential in CRF

2023年4月21日 星期五

13

The set of pixels in a detection d1 is denoted by Xd1 , yd1 represent the validity of detector

Energy function with detector potential

2023年4月21日 星期五

14

( ) ( ) ( , , )pix d d dd D

E X E x x H l

Pixel-based energy

Set of detections Detection score

Pixels in detection Detected Label

{0,1}

d

( , , ) min ( , , , )

y indicats whether detector hypothesis is validdd d d y d d d dx H l y x H l

2023年4月21日 星期五

15

{0,1}( , , ) min ( ( , ) ( , ) )

( , ) max(0,H ) define the strength of hypothesis

( , )( , ) = define the cost of label inconsistency

is detector threshold

N is t

dd d d d d d d d d

y

d d d d d t

d d dd d

d d

t

d

x H l f x H y g N H y

f x H w x H

f x H Ng N H

p x

H

d

henumber of inconsistent pixels

p is acceptable percentage of inconsistent pixels

Inference for detector potentials

Rewrite detector potential:

2023年4月21日 星期五

16

N

( , , ) min(0, ( , ) ( ))

( , ) min( ( , ), ( )).

Form of robust P potential can be soloved by graph

( ) min( , ( (

cut

( ), (

)

,

)

) ,

)

d

d

d d d d d d i di x

d d d d d i d

h max

i x

max l

l l il

i x

x H l f x H k x l

f x H f

x

x H k x

min k

l

f f l d

x l

'

' '

0

( ( , ) ( , ) ).d

d

d d d d d d dy

y

y arg min f x H y g N H y

Experimental Results

DatasetCamVid

10 minutes of high quality 30HZ

960 X 720 resolution

Three of four sequences shot in daylight, one shot in dusk

32 classes totally, 11 classes used in this papers

PASCAL VOC 200914743 images, 20 foreground class and 1 background class

749 training, 750 validation and 750 test images.

2023年4月21日 星期五

17

Details of CRF framework

Two level hierarchy CRF based on pixels and segmentsPixel-based potentials

Use TextonBoost to estimate the probability of a certain label by boosting weak classifiers based on a set of shape filter responses.

Segment-based potentialsSegments or super-segments based on Mean shift Joint Boosting algorithm

2023年4月21日 星期五

18

Detection-based potentials

DetectorsHistogram-based detector

Multiple features( bag of word, self-similarity, SIFT and oriented edges descriptors)Cascaded classifier composed of SVMs

Parts-based detectorHOG descriptorsDeformable parts and global templateLatent SVM

Output of detectorsBounding boxes with response scores

Foreground and background color modelGMM

2023年4月21日 星期五

19

Results on CamVid dataset

2023年4月21日 星期五

20

Result on PASCAL VOC dataset

2023年4月21日 星期五

21

2023年4月21日 星期五

22

Result on PASCAL VOC dataset

Summary

Integration of detectors with CRF.

Can handle occluded objects and false detections

Efficient and tractable with graph cut.

2023年4月21日 星期五

23

2023年4月21日 星期五

24

Thank you!

Any Question?