Contours and Optical Flow: Cues for Capturing Human Motion in Videos

Thomas Brox

Computer Vision and Pattern Recognition GroupUniversity of Bonn

Research partially funded by the German Research Foundation (DFG)

2

Thomas Brox

University of Bonn

Human pose tracking from video

1. Tracking of markers attached to the body+ Designed to be easy to track

Reliable and fast tracking

– Accuracy limited by number of markers

– People may feel uncomfortable

2. Tracking features that naturally appear in the images• Patches (e.g. KLT, SIFT, etc.)• Contour/Silhouette• Optic flow

How to extract these features reliably from the images

• Introduction

• Segmentation

• Optic Flow

• Summary

3

Thomas Brox

University of Bonn

Contour and optic flow based human tracking

Joint work with Bodo Rosenhahn

• Introduction

• Segmentation

• Optic Flow

• Summary

Part I

Object Contour Extraction

5

Thomas Brox

University of Bonn

Object contour extraction

• Find two regions: object & background

• Often: Static background background subtraction

• Optimality criteria here:– Strong similarity within regions– Small boundary

• Bayesian approach:

• Introduction

• Segmentation

• Optic Flow

• Summary

6

Thomas Brox

University of Bonn

Level set representation of contours(Dervieux-Thomasset 1979, Osher-Sethian 1988)

• Introduce embedding function • Contour C represented as zero-level line of

Courtesy of Daniel Cremers

• Introduction

• Segmentation

• Optic Flow

• Summary

7

Thomas Brox

University of Bonn

Region-based active contours (Chan-Vese 2001, Paragios-Deriche 2002)

• Minimize negative logarithm:

• Gradient descent:

plus update of p1 and p2

H(x)

H’(x)

• Introduction

• Segmentation

• Optic Flow

• Summary

8

Thomas Brox

University of Bonn

Region statistics

• 7 channels:– 3 color channels (CIELAB)– 4 texture channels

• Channels assumed to be independent

• Probability densities pij approximated by Gaussians

• Introduction

• Segmentation

• Optic Flow

• Summary

9

Thomas Brox

University of Bonn

Texture

• Usually modeled by Gabor filters(Gabor 1946)

• Includes1. Magnitude2. Orientation3. Scale

• High redundancy

• Sparse alternative representation feasible• Nonlinear structure tensor

(Brox et al. 2006)

• Region based local scale measure(Brox-Weickert 2004)

• Introduction

• Segmentation

• Optic Flow

• Summary

10

Thomas Brox

University of Bonn

Sparse texture features

Gabor filter bank Sparse representation

• Introduction

• Segmentation

• Optic Flow

• Summary

11

Thomas Brox

University of Bonn

Examples for contour extraction

• Introduction

• Segmentation

• Optic Flow

• Summary

12

Thomas Brox

University of Bonn

Local region statistics

• Object and background usuallynot homogeneous

• Idea: assume them to be locally homogeneous

• Probability densities estimated by local Gaussians

• Introduction

• Segmentation

• Optic Flow

• Summary

13

Thomas Brox

University of Bonn

Introducing a shape prior

• Idea: object model can serve as 3-D shape prior Constrains the segmentation, unwanted solutions

discarded

• Bayesian formula:

• Pose parameters of model unknown Two variables: contour and pose

• Introduction

• Segmentation

• Optic Flow

• Summary

14

Thomas Brox

University of Bonn

• Simultaneously optimize contour and pose:

• Iterative alternating scheme:– Update contour– Update pose parameters

• Related works: 2-D shape priors (Leventon et al. 2000, Cremers et al. 2002, Rousson-Paragios 2002)

Joint optimization

shape+pose constraint

conventional segmentation part

• Introduction

• Segmentation

• Optic Flow

• Summary

Part II

Optic Flow

16

Thomas Brox

University of Bonn

Optic flow based tracking

Image 1 and 2, estimate flow in between Given pose at Image 1

Estimated pose at Image 2Pose change due to optic flow

• Introduction

• Segmentation

• Optic Flow

• Summary

17

Thomas Brox

University of Bonn

Tracking example

• Introduction

• Segmentation

• Optic Flow

• Summary

18

Thomas Brox

University of Bonn

How to compute the optic flow?

• Given: two images I(x,y,t) and I(x,y,t+1) in a sequence

• Goal: displacement vector field (u,v) between these images

• Variational approach: (Horn-Schunck 1981)

• Introduction

• Segmentation

• Optic Flow

• Summary

19

Thomas Brox

University of Bonn

Enhanced model(Brox et al. 2004, Papenberg et al. 2006)

Robust smoothness term (Cohen 1993, Schnörr 1994)

Robust data term (Black-Anandan 1996, Mémin-Pérez 1996)

Spatiotemporal smoothness (Nagel 1990)

Gradient constancy (Brox et al. 2004)

Original Horn-Schunck:

Final optic flow model:

Non-linearized constancy (Nagel-Enkelmann 1986, Alvarez et al. 2000)• Introduction

• Segmentation

• Optic Flow

• Summary

20

Thomas Brox

University of Bonn

Horn-SchunckRobust smoothness

Impact of each improvement

Correct result Robust data termGradient constancyNonlinear constancySpatiotemporal smoothness

7.1

7

6.3

6

5.9

7

3.5

2.4

4

1.7

8

0

2

4

6

8 Horn-Schunck

Robust smoothness

Robust data term

Gradient constancy

Nonlinear constancy

Spatio-temporal smoothness

• Introduction

• Segmentation

• Optic Flow

• Summary

21

Thomas Brox

University of Bonn

Accurate and robust optic flow computation

Technique AAE

Nagel 10.22°

Uras et al. 8.94°

Alvarez et al. 5.53°

Mémin-Pérez 4.69°

Brox et al. (Noisy) 4.49°

Bruhn et al. 4.17°

Brox et al. 1.78°

• Introduction

• Segmentation

• Optic Flow

• Summary

22

Thomas Brox

University of Bonn

Contour and optic flow based human tracking

Joint work with Bodo Rosenhahn

• Introduction

• Segmentation

• Optic Flow

• Summary

23

Thomas Brox

University of Bonn

Summary

• Contours and optic flow can be reliable features for pose tracking

• Texture, local statistics, and a shape prior are important for general contour based human motion tracking

• High-end optic flow helps in case of fast motion

What’s next?

• Real-time performance• Automatic pose initialization• Prior knowledge about joint angle configurations

• Introduction

• Segmentation

• Optic Flow

• Summary

24

Thomas Brox

University of Bonn

Outlook

Joint work with Bodo Rosenhahn

• Introduction

• Segmentation

• Optic Flow

• Summary

25

Thomas Brox

University of Bonn

Backup: nonlinear structure tensor

• Texture orientation can be measured with the structure tensor (second moment matrix)(Förstner-Gülch 1987, Rao-Schunck 1991, Bigün et al. 1991)

• Gaussian smoothing nonlinear diffusion

Input image Linear structure tensor

Nonlinear structure tensor

• Introduction

• Segmentation

• Optic Flow

• Summary

26

Thomas Brox

University of Bonn

Backup: region based local scale measure

• Estimate regions, measuretheir size

• Nonlinear diffusion: TV flow(Andreu et al. 2001)

• Tends to yield piecewise constant images regions

• Local evolution speed inverselyproportional to size of region(Steidl et al. 2004)

local scale measure

• Introduction

• Segmentation

• Optic Flow

• Summary

Input image

Local scale