lecture'9'&'10:''...

Lecture 9 & 10 - Fei-Fei Li

Lecture'9'&'10:''Stereo'Vision'

Dr.'Juan'Carlos'Niebles'Stanford'AI'Lab'

'Professor'FeiAFei'Li'Stanford'Vision'Lab'

25AOctA16'1'


Point of observation

Figures © Stephen E. Palmer, 2002

Dimensionality'ReducIon'Machine'(3D'to'2D)'

3D world 2D image

25AOctA16'2'

Lecture 9 & 10 - Fei-Fei Li 25AOctA16'3'

Pinhole(camera(

• 'Common'to'draw'image'plane'in#front''of'the'focal'point'''• 'Moving'the'image'plane'merely'scales'the'image.'

f' f'

!!"

!!#

$

=

=

zy

f'y

zx

f'x


RelaIng'a'realAworld'point'to'a'point'on'the'camera'

In'homogeneous'coordinates:'

!!!!

"

#

$$$$

%

&

!!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

=

10100000000

'zyx

ff

zyfxf

P

M' ideal'world'

Intrinsic Assumptions •  Unit aspect ratio •  Optical center at (0,0) •  No skew

Extrinsic Assumptions •  No rotation •  Camera at (0,0,0)


RelaIng'a'realAworld'point'to'a'point'on'the'camera'

In'homogeneous'coordinates:'

P ' =f xf yz

!

"

####

$

%

&&&&

=

f 0 0 00 f 0 00 0 1 0

!

"

###

$

%

&&&

xyz1

!

"

####

$

%

&&&&

= K I 0[ ]P

Intrinsic Assumptions •  Unit aspect ratio •  Optical center at (0,0) •  No skew


K


RealAworld'camera'

!!!!

"

#

$$$$

%

&

!!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

10100000

10

0

zyx

vus

vu

w β

α


Intrinsic Assumptions •  Optical center at (u0, v0) •  Rectangular pixels •  Small skew

Intrinsic'parameters'

P ' = K I 0!"

#$P

Slide'inspiraIon:'S.'Savarese'

25AOctA16'6'


RealAworld'camera'+'RealAworld'transformaIon'

Extrinsic Assumptions •  Allow rotation •  Camera at (tx,ty,tz)

Intrinsic Assumptions •  Optical center at (u0, v0) •  Rectangular pixels •  Small skew

!!!!

"

#

$$$$

%

&

!!!

"

#

$$$

%

&

!!!

"

#

$$$

%

&

=

!!!

"

#

$$$

%

&

1100

01 333231

232221

131211

0

0

zyx

trrrtrrrtrrr

vus

vu

w

z

y

x

β

α

P ' = K R t[ ] P

Slide'inspiraIon:'S.'Savarese'

25AOctA16'7'


What'we'will'learn'today?'

•  IntroducIon'to'stereo'vision'•  Epipolar'geometry:'a'gentle'intro'

•  Parallel'images'&'image'recIficaIon'

•  Solving'the'correspondence'problem'

•  Homographic'transformaIon'

•  AcIve'stereo'vision'system'

'

25AOctA16'8'

Reading:('[HZ]'Chapters:'4,'9,'11''[FP]'Chapters:'10 ''








'

25AOctA16'9'



Recovering'3D'from'Images'

•  How'can'we'automaIcally'compute'3D'geometry'from'images?'– What'cues'in'the'image'provide'3D'informaIon?'

Point of observation

Real 3D world 2D image

?'

25AOctA16'10'


•  Shading'

Visual'Cues'for'3D'

Merle(Norman(Cosme5cs,(Los(Angeles(

Slide'cred

it:'J.'H

ayes'

25AOctA16'11'


Visual'Cues'for'3D'

•  Shading'

•  Texture'

The$Visual$Cliff,(by(William(Vandivert,(1960(

Slide'cred

it:'J.'H

ayes'

25AOctA16'12'


Visual'Cues'for'3D'

From(The$Art$of$Photography,(Canon(

•  Shading'

•  Texture'

•  Focus'

Slide'cred

it:'J.'H

ayes'

25AOctA16'13'


Visual'Cues'for'3D'

•  Shading'

•  Texture'

•  Focus'

•  MoIon'

Slide'cred

it:'J.'H

ayes'

25AOctA16'14'


Visual'Cues'for'3D'

•  Others:'–  Highlights'–  Shadows'–  Silhouedes'–  InterAreflecIons'–  Symmetry'

–  Light'PolarizaIon'–  ...'

•  Shading'

•  Texture'

•  Focus'

•  MoIon'Shape'From'X'

•  X'='shading,'texture,'focus,'moIon,'...'

•  We’ll'focus'on'the'moIon'cue'

Slide'cred

it:'J.'H

ayes'

25AOctA16'15'


Stereo'ReconstrucIon'

•  The'Stereo'Problem'

–  Shape'from'two'(or'more)'images'

–  Biological'moIvaIon'

known(camera(

viewpoints(

Slide'cred

it:'J.'H

ayes'

25AOctA16'16'


Why'do'we'have'two'eyes?'

Cyclops( ( (((((((((((( (vs.(((((((((((((((((((((( ((Odysseus(

Slide'cred

it:'J.'H

ayes'

25AOctA16'17'


1.'Two'is'beder'than'one'

Slide'cred

it:'J.'H

ayes'

25AOctA16'18'


Depth'from'Convergence'

Slide'cred

it:'J.'H

ayes'

25AOctA16'19'








'

25AOctA16'20'



Pinhole(camera(

• 'Common'to'draw'image'plane'in#front''of'the'focal'point'''• 'Moving'the'image'plane'merely'scales'the'image.'

f' f'

!!"

!!#

$

=

=

zy

f'y

zx

f'x


Triangula5on(

O' O’'

p’'

P'

p'

Inputs:'1.  Camera'parameters'known'2.  Corresponding'points'p'and'p’'known''

Output:'P'is'at'the'intersecIon'of'''''''''''and'

�!Op

��!O0p0


Epipolar(geometry(

• 'Epipolar'Plane' • 'Epipoles'e,'e’'

• 'Epipolar'Lines'• 'Baseline'

O' O’'

p’'

P'

p'

e' e’'

='intersecIons'of'baseline'with'image'planes''='projecIons'of'the'other'camera'center'


Example:(Converging(image(planes(


Epipolar(Constraint(

A 'Two'views'of'the'same'object''A 'Suppose'I'know'the'camera'posiIons'and'camera'matrices'A'Given'a'point'on'len'image,'how'can'I'find'the'corresponding'point'on'right'image?'


O' O’'

P?'

p'

e' e’'l’'



• ''PotenIal'matches'for'p'have'to'lie'on'the'corresponding'epipolar'line'l’.'

• ''PotenIal'matches'for'p’'have'to'lie'on'the'corresponding'epipolar'line'l.'



!!!

"

#

$$$

%

&

=→

1vu

PMp

O' O’'

p' p’'

P'

R,'T'

!!!

"

#

$$$

%

&'

'

='→

1vu

PMp

M = K I 0!"

#$ M ' = K ' R T!

"#$



O' O’'

p'p’'

P'

R,'T'

'''K'and'K’'are'known''''(calibrated'cameras)'

M = K I 0!"

#$ M ' = K ' R T!

"#$

[ ]0IM = [ ]TRM ='Assume'K'='K’'='I'



O' O’'

p'p’'

P'

R,'T'

[ ] 0)pR(TpT =!×⋅Perpendicular'to'epipolar'plane'

)( pRT !×


Cross'product'as'matrix'mulIplicaIon'

baba ][0

00

×=

"""

#

$

%%%

&

'

"""

#

$

%%%

&

'

−

−

−

=×

z

y

x

xy

xz

yz

bbb

aaaaaa

“skew'symmetric'matrix”'



O' O’'

p'p’'

P'

R,'T'

[ ] 0)pR(TpT =!×⋅ [ ] 0pRTpT =!⋅⋅→ ×

E'='essenIal'matrix'(LonguetAHiggins,'1981)'



•  E'p’''is'the'epipolar'line'associated'with'p’'(l'='E'p’)'•  ET'p''is'the'epipolar'line'associated'with'p'(l’'='ET'p)'•  E'is'singular'(rank'two)'•  E'e’'='0'''and'''ET'e'='0'•  E'is'3x3'matrix;'5'DOF''

O'O’'

p’'

P'

p'

e' e’'

l' l’'








'

25AOctA16'34'



Simplest Case: Parallel images •  Image planes of cameras

are parallel to each other and to the baseline

•  Camera centers are at same height

•  Focal lengths are the same

Slide'cred

it:'J.'H

ayes'

25AOctA16'35'


•  Image planes of cameras are parallel to each other and to the baseline

•  Camera centers are at same height

•  Focal lengths are the same •  Then, epipolar lines fall

along the horizontal scan lines of the images

Simplest Case: Parallel images

Slide'cred

it:'J.'H

ayes'

25AOctA16'36'


Essential matrix for parallel images

pTE !p = 0, E = [t× ]R

!!!

"

#

$$$

%

&

−== ×

0000

000][

TTRtE

!!!

"

#

$$$

%

&

−

−

−

=×

00

0][

xy

xz

yz

aaaaaa

a

R = I t = (T, 0, 0)

Epipolar constraint:

t

p

p�

Reminder:'skew'symmetric'matrix'

Slide'cred

it:'J.'H

ayes'

25AOctA16'37'


( ) ( ) vTTvvTTvuv

u

TTvu !==

"""

#

$

%%%

&

'

!

−="""

#

$

%%%

&

'!

!

)))

*

+

,,,

-

.

− 00

10100

00000

1

The y-coordinates of corresponding points are the same!

t

p

p�

Essential matrix for parallel images

pTE !p = 0, E = [t× ]R

!!!

"

#

$$$

%

&

−== ×

0000

000][

TTRtE

R = I t = (T, 0, 0)

Epipolar constraint:

Slide'cred

it:'J.'H

ayes'

25AOctA16'38'


Triangulation -- depth from disparity

f

p p�

Baseline B

z

O O�

P

f

disparity = u− "u = B ⋅ fz

Disparity is inversely proportional to depth!

Slide'cred

it:'J.'H

ayes'

25AOctA16'39'


Stereo image rectification

Slide'cred

it:'J.'H

ayes'

25AOctA16'40'


Algorithm: •  Re-project image planes onto a

common plane parallel to the line between optical centers

•  Pixel motion is horizontal after this transformation

•  Two transformation matrices, one for each input image reprojection

!  C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Stereo image rectification

Slide'cred

it:'J.'H

ayes'

25AOctA16'41'


Rectification example

25AOctA16'42'


Applica5on:(view(morphing(S.'M.'Seitz'and'C.'R.'Dyer,'Proc.#SIGGRAPH#96,'1996,'21A30''


Applica5on:(view(morphing(


Removing perspective distortion

Hp'

(rectification)

25AOctA16'47'








'

25AOctA16'48'



Stereo matching: solving the correspondence problem

•  Goal: finding matching points between two images

25AOctA16'49'


Basic stereo matching algorithm

•  For each pixel in the first image –  Find corresponding epipolar line in the right image –  Examine all pixels on the epipolar line and pick the best match –  Triangulate the matches to get depth information

•  Simplest case: epipolar lines are scanlines

–  When does this happen?

Slide'cred

it:'J.'H

ayes'

25AOctA16'50'


Basic stereo matching algorithm

•  If necessary, rectify the two stereo images to transform epipolar lines into scanlines

•  For each pixel x in the first image –  Find corresponding epipolar scanline in the right image –  Examine all pixels on the scanline and pick the best match x� –  Compute disparity x-x� and set depth(x) = 1/(x-x�)

corresponding'

25AOctA16'51'


•  Let�s make some assumptions to simplify the matching problem –  The baseline is relatively small (compared to the

depth of scene points) –  Then most scene points are visible in both views –  Also, matching regions are similar in appearance

Correspondence problem

Slide'cred

it:'J.'H

ayes'

25AOctA16'52'


Matching cost

disparity

Left Right

scanline

Correspondence search with similarity constraint

•  Slide a window along the right scanline and compare contents of that window with the reference window in the left image

•  Matching cost: SSD or normalized correlation

Slide'cred

it:'J.'H

ayes'

25AOctA16'53'


Left Right

scanline


SS

D (s

um o

f squ

ared

diff

eren

ces)

25AOctA16'54'


Left Right

scanline


Nor

mal

ized

cor

rela

tion

25AOctA16'55'


Effect of window size

– Smaller window +  More detail •  More noise

–  Larger window

+  Smoother disparity maps •  Less detail

W = 3 W = 20

25AOctA16'56'


The similarity constraint

•  Corresponding regions in two images should be similar in appearance

•  …and non-corresponding regions should be different •  When will the similarity constraint fail?

Slide'cred

it:'J.'H

ayes'

25AOctA16'57'


Limitations of similarity constraint

Textureless surfaces Occlusions, repetition

Specular surfaces Slide'cred

it:'J.'H

ayes'

25AOctA16'58'


Results with window search

Window-based matching Ground truth

Left image Right image

25AOctA16'59'


Better methods exist... (CS231a)

Graph cuts Ground truth

Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

25AOctA16'60'


The role of the baseline

• Small baseline: large depth error • Large baseline: difficult search problem

Large Baseline Small Baseline

Slide credit: S. Seitz

25AOctA16'61'


Problem for wide baselines: Foreshortening

•  Matching with fixed-size windows will fail! •  Possible solution: adaptively vary window size •  Another solution: model-based stereo (CS231a)

Slide credit: J. Hayes

25AOctA16'62'




•  Parallel'images'




'

25AOctA16'63'



Reminder:'transformaIons'in'2D'

Special'case'from'lecture'2'(planar'rotaIon''&'translaIon)'

u 'v '1

!

"

###

$

%

&&&= R t

0 1

!

"#

$

%&

uv1

!

"

###

$

%

&&&= He

uv1

!

"

###

$

%

&&&

-  3 DOF -  Preserve distance (areas) -  Regulate motion of rigid object

25AOctA16'64'


u 'v '1

!

"

###

$

%

&&&=

a1 a2 a3a4 a5 a6a7 a8 a9

!

"

####

$

%

&&&&

uv1

!

"

###

$

%

&&&= H

uv1

!

"

###

$

%

&&&

-  8 DOF -  Preserve colinearity

Reminder:'transformaIons'in'2D'

Generic'case'(rotaIon'in'3D,'scale''&'translaIon)'

25AOctA16'65'


Goal:'esImate'the'homographic'transformaIon'between'two'images'

H'

AssumpIon:'Given'a'set'of'corresponding'points.'


Goal:'esImate'the'homographic'transformaIon'between'two'images'

H'

AssumpIon:'Given'a'set'of'corresponding'points.'

QuesIon:'How'many'points'are'needed?'

Hint:'DoF'for'H?' 8!'

At'least'4'points'(8'equaIons)'


DLT algorithm (Direct Linear Transformation)

pi !pi

!pi = H pi

25AOctA16'68'

H


DLT algorithm (direct Linear Transformation)

!pi ×H pi = 0

9x1

0Ai =hFunction of measurements

Unknown [9x1]

[2x9]

2'independent'equaIons''

h =

h1h2h9

!

"

#####

$

%

&&&&&

25AOctA16'69'

H =

h1 h2 h3h4 h5 h6h7 h8 h9

!

"

####

$

%

&&&&



0hAi =9x2A

0hA1 =0hA2 =

0hAN =

0hA 199N2 =××

1x9h

Over determined Homogenous system

25AOctA16'70'

pi !pi

H



0hA 199N2 =××How to solve ?

Singular Value Decomposition (SVD)!

25AOctA16'71'



0hA 199N2 =××

999992 ××× Σ Tn VU

Last column of V gives h! H!

How to solve ?

Singular Value Decomposition (SVD)!

Why?'See'pag'593'of'AZ'

25AOctA16'72'



How to solve ? 0hA 199N2 =××

25AOctA16'73'


Clarification about SVD Tnnnnnmnm VDUP ×××× =

Tnnnmmmnm VDUP ×××× =

Has n orthogonal columns

Orthogonal matrix

•  This is one of the possible SVD decompositions •  This is typically used for efficiency •  The classic SVD is actually:

orthogonal Orthogonal

25AOctA16'74'


H'








'

25AOctA16'76'



Ac5ve(stereo((point)'

Replace'one'of'the'two'cameras'by'a'projector'A 'Single'camera''A 'Projector'geometry'calibrated'A 'What’s'the'advantage'of'having'the'projector?''

P'

O2'

p’'

Correspondence'problem'solved!'

projector'


Ac5ve(stereo((stripe)''

O'A Projector'and'camera'are'parallel'A 'Correspondence'problem'solved!'

projector'


Ac5ve(stereo((shadows)''

Light'source' O'

A '1'camera,1'light'source'A 'very'cheap'setup'A 'calibrated'light'source'

J.'Bouguet'&'P.'Perona,'99'S.'Savarese,'J.'Bouguet'&'Perona,'00''


Ac5ve(stereo((shadows)''J.'Bouguet'&'P.'Perona,'99'S.'Savarese,'J.'Bouguet'&'Perona,'00''


Ac5ve(stereo((colorAcoded'stripes)''

A'Dense'reconstrucIon'''A 'Correspondence'problem'again'A 'Get'around'it'by'using'color'codes'

projector' O'

L.'Zhang,'B.'Curless,'and'S.'M.'Seitz'2002'

S.'Rusinkiewicz'&'Levoy'2002''


Ac5ve(stereo((colorAcoded'stripes)''

L.'Zhang,'B.'Curless,'and'S.'M.'Seitz.'Rapid'Shape'AcquisiIon'Using'Color'Structured'Light'and'MulIApass'Dynamic'Programming.'3DPVT'2002'

Rapid'shape'acquisiIon:'Projector'+'stereo'cameras'


Ac5ve(stereo((stripe)''

•  OpIcal'triangulaIon'–  Project'a'single'stripe'of'laser'light'–  Scan'it'across'the'surface'of'the'object'–  This'is'a'very'precise'version'of'structured'light'scanning'

Digital Michelangelo Project http://graphics.stanford.edu/projects/mich/


Laser scanned models

The#Digital#Michelangelo#Project,'Levoy'et'al.'Slide credit: S. Seitz

25AOctA16'84'

Lecture 9 & 10 - Fei-Fei LiSlide credit: S. Seitz

25AOctA16'

The#Digital#Michelangelo#Project,'Levoy'et'al.'


85'




25AOctA16'86'




25AOctA16'87'


1.0'mm'resoluIon'(56'million'triangles)''

Slide credit: S. Seitz



25AOctA16'88'


What'we'have'learned'today'






'

25AOctA16'89'


lecture'9'&'10:''...

Documents