Lecture 04

22/11/2011Shai Avidan

לא: / המופיע זה ולא בכיתה הנלמד החומר הוא המחייב החומר הבהרה . במצגת מופיע

Epipolar Geometry

• Essential Matrix (Longuet-Higgins 1981)

• Fundamental Matrix (Faugeras 1992)

• F and Homographies

ESSENTIAL MATRIX

Stereo correspondence constraintsGeometry of two views allows us to constrain where the corresponding pixel for some image point in the first view must occur in the second view.

Epipolar constraint: Why is this useful?• Reduces correspondence problem to 1D search along conjugate

epipolar lines

epipolar planeepipolar lineepipolar line

Adapted from Steve Seitz

Epipolar geometry: termsBaseline: line joining the camera centers

Epipole: point of intersection of baseline with the image plane

Epipolar plane: plane containing baseline and world point

Epipolar line: intersection of epipolar plane with the image plane

All epipolar lines intersect at the epipole

An epipolar plane intersects the left and right image planes in epipolar lines

• Potential matches for p have to lie on the corresponding epipolar line l’.

• Potential matches for p’ have to lie on the corresponding epipolar line l.

Source: M. Pollefeys

http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html

Epipolar constraint

Example

Example: converging cameras

Figure from Hartley & Zisserman

As position of 3d point varies, epipolar lines “rotate” about the baseline

Example: motion parallel with image plane

Figure from Hartley & Zisserman

Example: forward motion

Figure from Hartley & Zisserman

e

e’

Epipole has same coordinates in both images.Points move along lines radiating from e: “Focus of expansion”

Stereo geometry, with calibrated cameras

If the rig is calibrated, we know :how to rotate and translate camera reference frame 1 to get to camera reference frame 2.

Rotation: 3 x 3 matrix; translation: 3 vector.

3d rigid transformation

TRXX

‘‘‘

Stereo geometry, with calibrated cameras

TRXX Camera-centered coordinate systems are related by known rotation R and translation T:

Cross product

Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs.

So here, c is perpendicular to both a and b, which means the dot product = 0.

From geometry to algebra

TRXX'

TTRXTXT

RXT

RXTXXTX

0Normal to the plane

Matrix form of cross product

Can be expressed as a matrix multiplication.

From geometry to algebra

TRXX'

TTRXTXT

RXT

RXTXXTX

0Normal to the plane

Essential matrix

0 RXTX

0 RXTX x

E is called the essential matrix, which relates corresponding image points [Longuet-Higgins 1981]

Let RTE x

0 EXX T

This holds for the rays p and p’ that are parallel to the camera-centered position vectors X and X’, so we have:

0Epp'T

Essential matrix and epipolar lines

pE T is the coordinate vector representing the epipolar line associated with point p’

Ep is the coordinate vector representing the epipolar line associated with point p

0EppEpipolar constraint: if we observe point p in

one image, then its position p’ in second image must satisfy this equation .

Essential matrix: propertiesRelates image of corresponding points in both cameras,

given rotation and translation

Assuming intrinsic parameters are known

E is a rank 2 matrix with two equal eigvenvalues

Given E, the camera projection matrix can be computed using SVD factorization. There are four possible solutions

RTE x

From E to Motion

FUNDAMENTAL MATRIX

Uncalibrated case

pp 1K''' 1 pp K

Camera coordinates

Internal calibration matrices, one per camera

Image pixel coordinates

pp K

So, for two cameras (left and right):

For a given camera:

Camera coordinates

Uncalibrated case: fundamental matrix

0' Epp From before, the essential matrix E.

0' 11 pEp KK

0'' 1 pEp KK TT

0' pFp T

Fundamental matrix

Fundamental matrix

Relates pixel coordinates in the two views

More general form than essential matrix: we remove need to know intrinsic parameters

If we estimate fundamental matrix from correspondences in pixel coordinates, can reconstruct epipolar geometry without intrinsic or extrinsic parameters

Computing F from correspondences

Cameras are uncalibrated: we don’t know E or left or right K matrices

Estimate F from 8+ point correspondences.

0' pFp T

1' KK T EF

Computing F from correspondencesEach point

correspondence generates one constraint on F

Collect n of these constraints

Solve for f , vector of parameters.

0' pFp T

F AND HOMOGRAPHIES

F decompositionGiven F, we want to decompose to left and right projection

matrices. Because reconstruction is up to global projective transformation we can set the left camera projection matrix to [I;0].

Since p’^tFp=0 for every pair of points, there is one point e’ s.t. e’^tFp=0 for every p -> e’ is in the null space of F.

e’ is the epipole and all epipolar lines go through it.

From the epipolar constraint (shown for the essential matrix) we have that

So, we have M_1=[I;0] and M_2 = [H,e’]

Where H is a homography

TT FnulleHeeHeF '0'''

The rank-4 of homographies

HeveeHeveHeGeF

GeF

veHG

HeF

TT

T

'''''''

:Proof

' : thathave We

'every for Then

'Let

:Lemma

The canonical homographyChoose some line l’ going through p’ then:

l’^T p’ = 0

And from the epipolar constraint we have:

p’^TFp = 0

Combining the two we have that p’ is on the intersection of the two lines:

P’ =~ [l’]_x Fp

And we have a homography equation: p’ =~ [l’]_x Fp =Hp

In particular, lets choose l’ =~ e’, which is guaranteed not to coincide with the epipolar lines (e’Te’ !=0) and have that H = [e’]_xF is a homography that is called the canonical homography matrix

Relationship between F and HWhat is the relationship between homography and

fundamental matrix?

Let’s choose some homography, then we have:

(Hp)^TFp = 0 and this is true for every point p.

So we have that Fp =~ [e’]_x Hp, because they both describe the epipolar line s.t. p’^TFp and p’^T [e’]_x Hp=0

Geometric interpretation:

The points Hp, p’ and e’ are all on the same line (the epipolar line)

And this can be written as:

p’^T[e]_xHp=0