cs 395/495-25: spring 2004

CS 395/495-25: Spring 2004CS 395/495-25: Spring 2004

IBMR:IBMR: The Fundamental MatrixThe Fundamental MatrixCombines Two CamerasCombines Two Cameras

Jack TumblinJack [email protected]@cs.northwestern.edu

Basic Camera PBasic Camera P00: P: P33PP22 ((oror camera camera RR33))

• Basic Camera PBasic Camera P0 0 is a 3x4 matrix:is a 3x4 matrix:

• Non-square pixels? change scaling Non-square pixels? change scaling ((xx, , yy)) • Parallelogram pixels? set nonzero skew Parallelogram pixels? set nonzero skew ss K matrixK matrix: “(internal) camera calib. matrix”: “(internal) camera calib. matrix”

xxyyzz

xxcc

yycc

zzcc

11

==

PP00 X = x X = x

yyy’y’

ffzzCC

yycc

xxcc

zzcc

pp

xxff ss ppxx 000 0 yyff ppyy 000 0 0 0 11 0 0

KK (3x3 submatrix)(3x3 submatrix)

[K 0] = P[K 0] = P00

ONEONE Camera Matrix P links P Camera Matrix P links P33PP22::• Basic cameraBasic camera: :

x = Px = P0 0 X X wherewhere PP00 = [K | 0] = = [K | 0] =

• World-space camera:World-space camera: translate X (origin) to camera location C, then rotate:translate X (origin) to camera location C, then rotate:

x = PX = (Px = PX = (P00·R·T) X = K·R·[·R·T) X = K·R·[II C] X C] X

• Rewrite as:Rewrite as:P = K[R -RC]P = K[R -RC]

• Redundant notation:Redundant notation:P = [M pP = [M p44]]M = RKM = RKpp44 = -K = -K··RR··CC

Output: Output: xx2D Camera Image2D Camera Image

Input:Input: XX3D World Space3D World Space

zz

yy

xx

XX (world space)(world space)

zz cc

yy cc

xx cc

CC

xx(c

amer

a (c

amer

a sp

ace)

spac

e)

~~

~~

xxff ss ppxx 000 0 yyff ppyy 000 0 0 0 11 0 0

~~

~~

ONE ONE Complete Camera Matrix PComplete Camera Matrix P• K matrixK matrix: “: “internalinternal camera calib. matrix” camera calib. matrix”• RR··T matrixT matrix: “: “externalexternal camera calib. matrix” camera calib. matrix”

– T matrix: Translate world to the cam. originT matrix: Translate world to the cam. origin– R matrix: 3D rotate world to fit cam. axesR matrix: 3D rotate world to fit cam. axes

Combine: writeCombine: write(P(P00··RR··TT··X = xX = x

or or K[R -RC]K[R -RC] ··X = x X = x oror PP··X = xX = x

Output: Output: xx2D Camera Image 2D Camera Image

Input:Input: XX3D World Space 3D World Space

zz

yy

xx

XX (world space)(world space)

zz cc

yy cc

xx cc

CC

xx(c

amer

a sp

ace)

(cam

era

spac

e)

~~CC

~~

~~CC

TWOTWO Cameras: Epipolar Geometry Cameras: Epipolar GeometryBasic idea:Basic idea:• Given Given ONLY imagesONLY images from 2 cameras from 2 cameras C, C’C, C’• Different views of same objects Different views of same objects XX, , butbut

we don’t knowwe don’t know world-space points world-space points XX. .

• If we choose an If we choose an xx, , how can we find how can we find x’x’ ? ?

• How are How are x, x’x, x’ linked? linked?CC

C’C’

x’x’xx

XX

TWOTWO Cameras: Epipolar Geometry Cameras: Epipolar GeometryBasic idea:Basic idea:• 2 cameras located at 2 cameras located at C, C’C, C’ in world space. in world space.• Find Find ‘baseline’‘baseline’ through camera centers through camera centers C, C’C, C’• Baseline hits image planes at Baseline hits image planes at ‘epipoles’‘epipoles’

Notice baseline and XNotice baseline and Xform a plane...form a plane...

(OTHER planes(OTHER planes thru thru baseline for other X...)baseline for other X...) baselinebaseline

CCC’C’

x’x’xx

XX

XX

TWOTWO Cameras: Epipolar Geometry Cameras: Epipolar GeometryBasic idea:Basic idea:• 2 cameras located at C, C’ in world space.2 cameras located at C, C’ in world space.• Find ‘baseline’ through camera centers C, C’Find ‘baseline’ through camera centers C, C’• Baseline hits image planes at ‘epipoles’Baseline hits image planes at ‘epipoles’• Family of planes thru baselineFamily of planes thru baseline

are all the are all the ‘epipolar planes’‘epipolar planes’• Image of planes = lines =Image of planes = lines =

‘epipolar lines’‘epipolar lines’• Lines intersect at Lines intersect at

epipolar pointsepipolar points in both images. in both images.CC

C’C’baselinebaseline

x’x’xx

TWOTWO Cameras: Epipolar Geometry Cameras: Epipolar Geometry

Summary:Summary:• Connect cameras Connect cameras C, C’C, C’ with a with a baselinebaseline, which, which

hits image planes at hits image planes at epipolesepipoles e, e’. e, e’. • Chose any world pt Chose any world pt X, X, then then everything is coplanar!everything is coplanar!

epipolar planeepipolar plane includes includes image pointsimage points x, x’, and x, x’, and thesethese connect to connect to epipolesepipoles e,e’ by e,e’ by epipolar linesepipolar lines L, L’ L, L’

baselinebaselineCC C’C’

epipolar line L

epipolar line L epipolar line L

’

epipolar line L

’epipolar plane epipolar plane XX

x’x’

e’e’

xx

ee

Epipolar GeometryEpipolar Geometry

Useful properties:Useful properties:• Every image point Every image point xx maps to an epipolar line maps to an epipolar line L’L’

also:also:– Epipoles Epipoles e,e’e,e’ = each camera’s view of the other = each camera’s view of the other – All epipolar linesAll epipolar lines L’ L’ pass through epipole pass through epipole e’e’– Epipolar Line Epipolar Line L’L’ is (image of the is (image of the CCXX ray...) ray...)– Epipolar Line Epipolar Line L’L’ links (image of links (image of CC) to (image of ) to (image of XX))


epipolar line L


’

epipolar line L

’epipolar plane epipolar plane XX

x’x’

e’e’

xx

ee

Fundamental Matrix: Fx = L’Fundamental Matrix: Fx = L’

One Matrix Summarizes ALL of Epipolar Geometry!One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix Fundamental Matrix FF: 3x3, rank 2.: 3x3, rank 2.• Maps image point Maps image point xx to image point to image point x’:x’: x’ x’TT F x = 0 F x = 0

but F is only Rank 2 – but F is only Rank 2 – given only x, F given only x, F cannot cannot find x’ for you!! find x’ for you!!

• Maps image point Maps image point xx to epipolar line to epipolar line L’L’: : F x = L’F x = L’


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx

ee

epipolar plane epipolar plane


One Matrix Summarizes ALL of Epipolar Geometry!One Matrix Summarizes ALL of Epipolar Geometry! Fundamental Matrix Fundamental Matrix FF: 3x3, rank 2.: 3x3, rank 2.• Maps image point x to image point x’:Maps image point x to image point x’: (x’ (x’TT F) x = 0 F) x = 0

(We know x is on line L, thus (L(We know x is on line L, thus (LTT) x = 0, so) x = 0, so

• Maps image point Maps image point xx to epipolar line to epipolar line L’L’: : F x = L’F x = L’


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx

ee



• 2) What is 2) What is FF if we DON’T know the cameras if we DON’T know the cameras P, P’,P, P’,but we DO know some corresp. point pairs (x, x’)?but we DO know some corresp. point pairs (x, x’)?– FF finds epipolar line finds epipolar line L’L’ from point from point x x : Fx = L’: Fx = L’– (Recall that if (any) point (Recall that if (any) point x’x’ is on line a is on line a L’L’, then , then x’x’TT L’= 0 L’= 0) ) – Substitute Substitute FxFx for L’: for L’: x’x’TTF x = 0F x = 0AHA! we can find F using DLT-like method! AHA! we can find F using DLT-like method! see Chap. 10see Chap. 10


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

Derive the Fundamental Matrix:Derive the Fundamental Matrix:

– Recall Pseudo-Inverse:Recall Pseudo-Inverse: P P++ = P = PT T ( P P( P PTT))-1-1 and and P PP P++= I = I (pg148)(pg148)

– Write world-space ray Write world-space ray XXCC as: as: X(X() = P) = P++x + x + C C – Other camera’s image of the ray is its epipolar line Other camera’s image of the ray is its epipolar line L’L’: :

L’(L’() = P’X() = P’X() = P’P) = P’P++x + x + P’CP’C ..– But But P’C = e’P’C = e’; it is the epipole of the other camera, so; it is the epipole of the other camera, so

L’(L’() = P’P) = P’P++x + x + e’e’


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())


– But But P’C = e’P’C = e’; it is the epipole of the other camera, so; it is the epipole of the other camera, so

L’(L’() = P’P) = P’P++x + x + e’e’ ..– Rewrite epipolar line L’ using points Rewrite epipolar line L’ using points P’PP’P++xx and and e’ e’ – (Recall: find line between two P(Recall: find line between two P22 points with cross product: L= x points with cross product: L= x11 x x22))

– To get To get L’ = e’ L’ = e’ PP’ ’ PP+ + x x or or L’ = F x L’ = F x


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())


– But But P’C = e’P’C = e’; it is the epipole of the other camera, so; it is the epipole of the other camera, so

L’(L’() = P’P) = P’P++x + x + e’e’ ..– Rewrite epipolar line L’ using points Rewrite epipolar line L’ using points P’PP’P++xx and and e’ e’ – (Recall: find line between two P(Recall: find line between two P22 points with cross product: L= x points with cross product: L= x11 x x22))

– To get To get L’ = (e’) L’ = (e’) (P(P’ ’ PP+ + x) x) or or L’ = F x L’ = F x

THUS THUS F = [e’] F = [e’] PP’ ’ PP++


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

Hunh? What Is This?!?!Hunh? What Is This?!?!


F = [e’]F = [e’] PP’ ’ PP++ But what’s this? A NEW TRICK:But what’s this? A NEW TRICK:– Cross Product written as matrix multiply Cross Product written as matrix multiply (Zisserman pg. 554)(Zisserman pg. 554)

a a b =b = = = = [a] = = = [a] · · b b

– So write: So write: a a b = -b b = -b a = [a]a = [a]··b = (ab = (aTT··[b][b]))TT


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

aa11

aa22

aa33

bb11

bb22

bb33

aa22bb33 – a – a33bb22

aa33bb11 – a – a11bb33

aa11bb22 – a – a22bb11

0 -a0 -a33 a a22

a a33 0 -a 0 -a11

-a-a22 a a11 0 0

bb11

bb22

bb33a ‘skew symmetric’ a ‘skew symmetric’ matrixmatrix


• Given a point pair (x,x’) Matrix F is unique!Given a point pair (x,x’) Matrix F is unique!• Cool! still works even for different cameras!Cool! still works even for different cameras!• F tied DIRECTLY to corresp. point pairs (x, x’):F tied DIRECTLY to corresp. point pairs (x, x’):

– FF finds epipolar line finds epipolar line L’L’ from point from point x x : Fx = L’: Fx = L’– (Recall that if (any) point (Recall that if (any) point x’x’ is on line a is on line a L’L’, then , then x’x’TT L’= 0 L’= 0) ) – Substitute Substitute FxFx for L’: for L’: x’x’TTF x = 0F x = 0


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

F = [e’] F = [e’] PP’ ’ PP++


1) How do we find1) How do we find FF?? If we know the camera matrices If we know the camera matrices PP and and P’P’(we almost never do), we showed (we almost never do), we showed (Zisserman pg 224)(Zisserman pg 224)

F = [e’]F = [e’] PP’ ’ PP++


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

(Recall: P(Recall: P+ = + = PPTT(PP(PPTT))-1-1, the pseudo-inverse), the pseudo-inverse)

( Recall: a ( Recall: a b = -b b = -b a = [a]a = [a]··b = (ab = (aTT··[b][b]))T T ))


• 2) What is 2) What is FF if we DON’T know the cameras if we DON’T know the cameras P, P’,P, P’,but we DO know some corresp. point pairs (x, x’)?but we DO know some corresp. point pairs (x, x’)?– FF finds epipolar line finds epipolar line L’L’ from point from point x x : Fx = L’: Fx = L’– (Recall that if (any) point (Recall that if (any) point x’x’ is on line a is on line a L’L’, then , then x’x’TT L’= 0 L’= 0) ) – Substitute Substitute FxFx for L’: for L’: x’x’TTF x = 0F x = 0AHA! we can find F using DLT-like method! AHA! we can find F using DLT-like method! see Chap. 10see Chap. 10


epipolar line L


’

epipolar line L

’

XX

x’x’

e’e’

xx


X(X())

Fundamental Matrix SummaryFundamental Matrix SummaryFF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF

• If world space pt If world space pt XX image space pts. image space pts. x x and and x’x’ then then

x’x’TTF x = 0F x = 0

• Every image pt has epipolar line in the other image:Every image pt has epipolar line in the other image:

Fx = L’Fx = L’ FFTTx’ = Lx’ = L

• Baseline pierces image planes at epipoles e, e’ :Baseline pierces image planes at epipoles e, e’ :

Fe = 0Fe = 0 FFTTe’ = 0e’ = 0

(pg. 226)(pg. 226)

Fundamental Matrix SummaryFundamental Matrix SummaryFF is 3x3 matrix, maps is 3x3 matrix, maps PP22PP22, rank 2, 7-DOF, rank 2, 7-DOF

• Given camera matrices Given camera matrices PP, , P’P’, find , find FF matrix by: matrix by:F = [e’]F = [e’] P’ PP’ P++ (recall: e’ is image of C: e’ = P’C)(recall: e’ is image of C: e’ = P’C)

• FF is unaffected by any proj. transforms done is unaffected by any proj. transforms done on BOTH camerason BOTH cameras

((PHPH, , P’HP’H) has same F matrix as () has same F matrix as (PP, , P’P’) for ) for anyany full-rank full-rank HH(e.g. F measures camera C vs. Camera C’ only, (e.g. F measures camera C vs. Camera C’ only, no matter where you put them)no matter where you put them)

(pg. 226)(pg. 226)

Fundamental Matrix UsesFundamental Matrix UsesSpecial case: camera translate only Special case: camera translate only (no rotations)(no rotations)

• Camera matrices are Camera matrices are P= K[ P= K[ I | 0 I | 0 ]], , P’ = K[P’ = K[I | t I | t ]]– where where KK is internal calib., is internal calib., tt is 3D translation vector is 3D translation vector

• F matrix simplifies to F matrix simplifies to F = [e’]F = [e’]• Epipolar lines are all parallel to direction Epipolar lines are all parallel to direction tt• x,x’x,x’ displacement depends only on displacement depends only on tt & 3D depth & 3D depth zz::

x’ = x + (Kx’ = x + (Ktt)(1/z))(1/z)

ttxx

ttyy

ttzz

Fundamental Matrix UsesFundamental Matrix UsesCan we find a camera matrix Can we find a camera matrix KK from motion + fundamental from motion + fundamental FF??

• Let one camera position define the world’s coords:Let one camera position define the world’s coords:P = PP = P00 = K[ = K[ I | 0 I | 0 ]], , and define the other as and define the other as

P’ = [M | m] = K[R P’ = [M | m] = K[R | - -RC] RC] where where KK is internal calib., is internal calib., R R is rotation is rotation CC is world-space position is world-space position

• FF matrix simplifies to matrix simplifies to F = F = [[mm]]M M • If we know how we moved the camera (If we know how we moved the camera (R, C matricesR, C matrices) )

then find then find FF by correspondence, and then solve for K. by correspondence, and then solve for K. (pg 237) (pg 237)

~~

~~~~

Fundamental Matrix UsesFundamental Matrix UsesGeneral movement?General movement?• Recall:Recall: rotations don’t change image content rotations don’t change image content

(camera rotate(camera rotate a P a P3 3 homography matrix H)homography matrix H)• ANYANY cameras, cameras, ANYANY movements can then be movements can then be

warped to remove rotations, warped to remove rotations, THUSTHUS• Can Can ALWAYSALWAYS get parallel epipolar lines! get parallel epipolar lines!

– Easier to find correspondencesEasier to find correspondences– Easier to find depth values zEasier to find depth values z– ‘‘Parallel Epipolar Lines’==‘Rectified Image Pair’ Parallel Epipolar Lines’==‘Rectified Image Pair’

Fundamental Matrix PropertiesFundamental Matrix PropertiesWhy bother with F?Why bother with F?• Can find it from image pt. correspondences onlyCan find it from image pt. correspondences only• Works even for mismatched cameras Works even for mismatched cameras

(example: 100-year time-lapse of Eiffel tower)(example: 100-year time-lapse of Eiffel tower)• Choose your own world-space coordinate system.Choose your own world-space coordinate system.• SVD lets us recover P, P’ camera matrices from FSVD lets us recover P, P’ camera matrices from F

– (4-way ambiguity; what is front/back of C and C’?) (4-way ambiguity; what is front/back of C and C’?) pg 240pg 240

– BUT WE DON’T NEED TO!BUT WE DON’T NEED TO!• Complete 2-camera mapping from worldComplete 2-camera mapping from worldimageimage

– 2 images + corresponding point pairs (x2 images + corresponding point pairs (xii,x’,x’ii))FF

– Let camera coords == 3D world coords, then Let camera coords == 3D world coords, then (x(xii,x’,x’ii))XXii

(pg. 226)(pg. 226)

Correspondence Problem:Correspondence Problem:Where Computer Vision, IBMR part ways: Where Computer Vision, IBMR part ways:

Few Images:Few Images: (image pairs, trios…) (image pairs, trios…)FIND Fundamental Matrix + FIND Fundamental Matrix +

FIND Corresponding point pairs (x,x’) FIND Corresponding point pairs (x,x’)

************Versus**************************Versus************** Many Images:Many Images:

GATHER enough for P2-only work: GATHER enough for P2-only work: ignore or simplify the correspondence?ignore or simplify the correspondence?

ConclusionsConclusions• PP22, P, P33 matrix forms give elegant, principled matrix forms give elegant, principled

notation for ALL image geometrynotation for ALL image geometry– Cameras, lights, points, lines, planes, conics, Cameras, lights, points, lines, planes, conics,

quadrics, twisted cubics, …quadrics, twisted cubics, …– Matrix form makes everything reversible: 3D Matrix form makes everything reversible: 3D

from (2D)*!from (2D)*!– We can find shapes from images,We can find shapes from images,

BUT ONLY if we have enough BUT ONLY if we have enough reliable point correspondences…reliable point correspondences…And THAT’s what we usually CAN’T find!And THAT’s what we usually CAN’T find!

ENDEND

Another way to describe P3 lines!Another way to describe P3 lines!• Given world-space camera position Given world-space camera position

C =[a b c 1] and C =[a b c 1] and • a world-space direction D (found from a pixel xa world-space direction D (found from a pixel xdd, as shown , as shown

on prev. slide), on prev. slide), • even though D is a ‘point at infinity’, we can use it to make a even though D is a ‘point at infinity’, we can use it to make a

parametric line in world space:parametric line in world space:line(t) = C + Dt =[a b c 1] + [d e f 0]tline(t) = C + Dt =[a b c 1] + [d e f 0]t

• Note that ‘t’ is position along the line, and it’s in world-space Note that ‘t’ is position along the line, and it’s in world-space coords!coords!

• If you have two cameras C0, C1, and found two If you have two cameras C0, C1, and found two corresponding image points p0 and p1, you can use method corresponding image points p0 and p1, you can use method on previous slide to find a world-space direction D0, D1 for on previous slide to find a world-space direction D0, D1 for each image point, then make two parameterized lines each image point, then make two parameterized lines L0(t0), L1(t1) and solve for their world-space intersection L0(t0), L1(t1) and solve for their world-space intersection point X. THAT’S how you can do triangulation!.point X. THAT’S how you can do triangulation!.

cs 395/495-25: spring 2004

Documents