18 videotexture

7/30/2019 18 VideoTexture

1/61

Video Texture

15-463: Computational PhotographyAlexei Efros, CMU, Fall 2005

A.A. Efros


2/61

Weather Forecasting for Dummies

Lets predict weather: Given todays weather only, we want to know tomorrows

Suppose weather can only be {Sunny, Cloudy, Raining}

The Weather Channelalgorithm: Over a long period of time, record:

How often S followed by R

How often S followed by S

Etc.

Compute percentages for each state:

P(R|S), P(S|S), etc.

Predict the state with highest probability!

Its a Markov Chain


3/61

4.04.02.0

3.03.04.0

1.06.03.0

Markov Chain

What if we know today and yestardays weather?


4/61

Text Synthesis

[Shannon,48] proposed a way to generateEnglish-looking text using N-grams:

Assume a generalized Markov model

Use a large text to compute prob. distributions ofeach letter given N-1 previous letters

Starting from a seed repeatedly sample this Markovchain to generate new letters

Also works for whole words

WE NEED TO EAT CAKE


5/61

Mark V. Shaney (Bell Labs)

Results (using al t . si ngl es corpus):

As I've commented before, really relating tosomeone involves standing next to

impossible.

One morning I shot an elephant in my armsand kissed him.

I spent an interesting evening recently with

a grain of salt


6/61

Video TexturesVideo Textures

Arno Schdl

Richard Szeliski

David Salesin

Irfan Essa

Microsoft Research, Georgia Tech


7/61

Still photosStill photos


8/61

Video clipsVideo clips


9/61

Video texturesVideo textures


10/61

Problem statementProblem statement

video clip video texture


11/61

Our approachOur approach

How do we find good transitions?


12/61

Finding good transitionsFinding good transitions Compute L2 distance Di,j between all

frames

Similar frames make good transitions

frame ivs.

framej


13/61

Markov chain representationMarkov chain representation

2 3 41

Similar frames make good transitions


14/61

Transition costsTransition costsTransition from i to j if successor of i is similar

to j Cost function: Cij =Di+1,j

i

j

i+1

j-1i j D

i+1, j


15/61

Transition probabilitiesTransition probabilitiesProbability for transition P ij inversely related

to cost: Pij ~exp ( Cij / 2 )

high

low


16/61

Preserving dynamicsPreserving dynamics


17/61



18/61


Cost for transition ij

Cij = wk Di+k+1,j+kk = -N

N-1

i

j j+1

i+1 i+2

j-1j-2

i jDi, j-1 D Di+1, j i+2, j+1

i-1

Di-1, j-2


19/61

Preserving dynamics effectPreserving dynamics effect

Cost for transition ij

Cij = wk Di+k+1,j+kk = -N

N-1


20/61

2 3 41

Dead endsDead ends

No good transition at the end of sequence


21/61

2 3 41

Future costFuture cost

Propagate future transition costs backward

Iteratively compute new cost Fij =Cij +mink Fjk


22/61

2 3 41





23/61

2 3 41





24/61

2 3 41





25/61

2 3 41



Q-learning



26/61

Future cost effectFuture cost effect


27/61

Finding good loopsFinding good loops

Alternative to random transitions

Precompute set of loops up front


28/61

Visual discontinuitiesVisual discontinuities

Problem: Visible J umps


29/61

CrossfadingCrossfading

Solution: Crossfade from one sequence to

the other.

Ai-2

Ai-2

Bj-2

15

3

1 2

2 1

3

4

4 4

4 4

4+ + +

Ai-1

Ai-1/Bj-2 Ai-1/Bj-2 Ai-1/Bj-2

Bj-1

Ai

Bj

Ai+1

Bj+1

Bj+1


30/61

MorphingMorphing

Interpolation task:

A2

5 B2

5 C1

5+ +


31/61

MorphingMorphing

Interpolation task:

Compute correspondencebetween pixels of all frames

A2

5 B2

5 C1

5+ +


32/61

MorphingMorphing

Interpolation task:

Compute correspondencebetween pixels of all frames

Interpolate pixel position andcolor in morphed frame

based on [Shum 2000]

A2

5 B2

5 C1

5+ +


33/61

Results crossfading/morphingResults crossfading/morphing


34/61

Results crossfading/morphingResults crossfading/morphing

J ump Cut Crossfade Morph


35/61

CrossfadingCrossfading


36/61

Frequent jump & crossfadingFrequent jump & crossfading


37/61

Video portraitVideo portrait

Useful for web pages


38/61

Region-based analysisRegion-based analysis

Divide video up into regions

Generate a video texture for each region


39/61

Automatic region analysisAutomatic region analysis


40/61

Video-based animationVideo-based animation

Like sprites

computer games Extract sprites

from real video

Interactively controldesired motion

1985 Nintendo of America Inc.


41/61

Video sprite extractionVideo sprite extraction

blue screen matting

and velocity estimation


42/61

Ci j = + angle Ci j

vector tomouse pointer

Similarity term Control term

velocity vector

Animation

Video sprite controlVideo sprite control

Augmented transition cost:


43/61

Interactive fishInteractive fish


44/61

Lord of the FliesLord of the Flies


45/61

SummarySummary

Video clips video textures

define Markov process preserve dynamics avoid dead-ends

disguise visual discontinuities

i l i h i f


46/61

Motion Analysis & Synthesis [Efros 03]

What are they doing?

Activity recognition, surveillance, anti-terrorism

Can we do the same?

Motion retargeting, movies, video games, etc.

G h i i d


47/61

Gathering action data

Low resolution, noisy data

Moving camera

Occlusions

Fi t i R t ti


48/61

Figure-centric Representation

Stabilized spatio-temporal

volume

No translation information

All motion caused by persons

limbs

Good news: indifferent to camera

motion

Bad news: hard!

Good test to see if actions, not

just translation, are being

captured

R b f Thi P t


49/61

input sequence

Remembrance of Things Past

Explain novel motion sequence with bitsand pieces of previously seen video clips

Challenge: how to compare motions?

motion analysisrun

walk leftswing

walk right

jog

motion synthesis

synthetic sequence

H t d ib ti ?


50/61

How to describe motion?

Appearance

Not preserved across different clothing

Gradients (spatial, temporal) same (e.g. contrast reversal)

Edges

Too unreliable

Optical flow

Explicitly encodes motion Least affected by appearance

but too noisy

M ti D i t


51/61

Motion Descriptor

Image frame Optical flow yxF,

yx FF , ++ yyxx FFFF ,,, blurred ++ yyxx FFFF ,,,

Comparing motion descriptors


52/61

Comparing motion descriptors

t

motion-to-motion

similarit matrixblurry I

frame-to-frame

similarit matrix

I matrix

Recognizing Tennis


53/61

Recognizing Tennis

Red bars show classification results

Do as I Do Motion Synthesis


54/61

Do as I Do Motion Synthesis

Matching two things:

Motion similarity across sequences

Appearance similarity within sequence

Dynamic Programming

input sequence

synthetic sequence

Smoothness for Synthesis


55/61

Smoothness for Synthesis

is similarity between source and target frames

is appearance similarity within target frames

For every source frame i, find best target frame

by maximizing following cost function:

Optimize using dynamic programming

appW

actW

)1,(),(2

1

1

=

=

++n

i

iiappapp

n

i

iactact WiW

i

Do as I Do


56/61

Do as I DoSource Motion Source Appearance

Result

3400 Frames

Do as I Say Synthesis


57/61

Do as I Say Synthesis

Synthesize given action labels

e.g. video game control

run walk left swing walk right jog

synthetic sequence

run

walk leftswing

walk rightjog

Do as I Say


58/61

Do as I Say

Red box shows when constraint is applied

Application: Motion Retargeting


59/61

Application: Motion Retargeting

Rendering new character into existing

footage

Algorithm

Track original character

Find matches from new character Erase original character

Render in new character

Need to worry about occlusions

Context-based Image Correction


60/61

Context based Image Correction

Input sequence

3 closestframes

median images

Actor Replacement


61/61

Actor Replacement

SHOW VIDEO

18 videotexture

Documents