视觉的三维运动理解刘允才

上海交通大学

2002 年 11 月 16 日

Understanding 3D Motion

from Images

Yuncai Liu

Shanghai Jiao Tong University

November 16, 2002

Understanding motions and structures of 3D scene

A basic problem of visual system and computer vision

3-D Motion

Non-rigid objectmotion

Articulated objectmotion

Rigid objectmotion

3D rigid motion understanding

Use 3D Features Use 2D Features

Monocular Images Binocular Images

Stereo Structured light

Points Lines Corners

Points Lines

Texture

Minimum Number of Correspondences for Motion Solution with Monocular Images

Correspondences Min Number Degeneration

3D-3D points 3 collinear

3D-3D lines 2 parallel

2D-3D points 3

2D-3D lines 3

2D-2D points 5 quadratic surface

2D-2D lines 6 (3 frames)

2D corners 1 CC + 2 PC

xo

y

o X

YP (X, Y)

P (x, y, z)

L

l

N

P0

focus length = f

r

R

Perspective Projection

z

yfY

z

xfX ,

p

Point:

pz

P 1

Line: AX + BY + C = 0

( , , )N A B C

N p l

is normal of projection plan !

3D motion expression

Motion of a rigid object in 3D is usually expressed as a rotation around system origin followed by a translation.

Let

z

y

x

p

be a 3D point of an object at time t1

z

y

x

p

be a 3D point of an object at time t2

R be rotation , a 3x3 orthonormal matrix.

z

y

x

T

T

T

T

be translation vector.

Then, from time t1 to t2 :

TpRp

)3()2()1(

333231

232221

131211

|| rrr

rrr

rrr

rrr

R

kiirrr kii ,3,2,1,0,1 )()(2)(

Note:

There are only 3 independent parameters in R.

Properties of rotation matrix

11 RRRRorRR TTT

UUR

VUVRUR

)( VURVRUR

R is expressed by rotation axis and rotation angle.

Let rotation axis be

where: n12+n2

2+n32=1.

Rotation is made by a rotating with an angle around a rotation axis.

3

2

1

ˆ

n

n

n

n

sin)cos1( 32121 nnnr

cos)1( 22

2222 nnr

sin)cos1( 13223 nnnr sin)cos1( 23131 nnnr sin)cos1( 13232 nnnr

cos)1( 23

2333 nnr

sin)cos1( 23113 nnnr sin)cos1( 32112 nnnr

cos)1( 21

2111 nnr

Then the elements of R are:

Express R by 3 rotation angles.

CCr 11

SCCSSr 12

SSCSCr 13

SCr 21

CCSSSr 22

κωκφω CSSSCr 23

Sr 31

CSr 32

CCr 33

Quaternion form of rotation:

Quaternion is a four-element vector,

which can be used to express a rotation:

Let a rotation around axis by angle

;,,, 3210 qqqqq 123

22

21

20 qqqq

2

22

12

32

010322031

10322

32

12

22

03021

203130212

32

22

12

0

)(2)(2

)(2)(2

)(2)(2

)(

qqqqqqqqqqqq

qqqqqqqqqqqq

qqqqqqqqqqqq

qR

3

2

1

ˆ

n

n

n

n

Quaternion product:

A 3D vector can be expressed as a quaternion with scalar part being zero:

Pure rotation in 3D:

Express the rotation by quaternion:

uuuququuqqqq 0000 ,

pkzjyixr

00

kzjyixp

pRp

qrqr

Motion from 2D PC

------- From 2D images to determine 3D motion.

------- At least 5-point correspondences over two- image view are required.

------- 3D translation can only be determined over a scale factor.

------- Degeneration case: 3D points are on a quadratic surface.

Assume:

------ Single stationary camera.

------ Central projection model.

------ Rigid moving object.

------ Focus length f = 1, thus

; z

y = Y ;

z

x = X

p z

1 = )1 Y, (X, = P T

Let 3D motion from of time to of

(1)

Where

From equ (1)

(2)Apply to both sides of equ (2)

(3)

p p

T + R

p = p

2t1t

Tzyx TTTT

rrr

rrr

rrr

R

;

,333231

232221

131211

T + R

Pz = P z T

P Tz = P T z

R

Apply to both sides of equ (3)

(4)Let we define

(5)The eq(4) can be rewritten as

(6)

Note: eq.(6) is a homogeneous scalar equation. is a matrix containing only motion parameters, 8 or more PCs can uniquely determine E, subject to:

P

0 = P R

T P

] R T | R T | R T [ = RT = E (3)(2)(1)

0 = P E

P

RT = E

33

22 E

R

pT

Rp

p’

PP’

RP

o 0 PRTP

After matrix E is found ,translation can be solved:

i.e.

(7)

T

0 = T R T = T E

0 = T ET

can be determined from eq(7) subject to T

12 T

Once is obtained, rotation R can be obtained by least-square method:

(8)

Or let

T

0 R - E = T

][

][321

321

R | R | R R =

E | E | E E = )()()(

)()()(

][ = 213132321 EE + TE | EE + TE | EE + TER )()()()()()()()()(

Note, 180o reflection of motion is still a solution of equ (7) (homogeneous equation). In this case, object is moving behind the camera. To check for a real solution, we apply to both sides of equ (2).

Therefore if z > 0, it must hold that

Thus if,

let

P

0 = T P + P R Pz

0 > )P T( )P R

P(

0 < )P T( )P R

P(i

T - T

Motion from LC’s

------ from two image frames of a single camera,

3D motion can never be solved.

------ over 3 frames, at least 6 LC’s are required.

------ motion models

------ for a linear algorithm, 13 LC’s are needed.

n R = n

n R = nT + p R = pT + p R = p

23

12

2323

1212

Model A

Model B

Relation between model A and B:

n R = n

n R = nT + p R = pT + p R = p

13

12

1313

1212

T + T R = T , R R = R

23122313231213

xo

y

o X

Y

L

l

P0

focus length = f

N

N’ R-1

L’

l’

For a pure rotation,

and

are collinear.

In case of three frames, three collinear vectors collinear, forming a null parallelepiped, i.e.,

NR

1 N

0)( 113

112 NRNRN

o

o

N’

L’

l l’

N

For a pure translation,

and lie in a plan that pass though origin and perpendicular to 3D line l .

Thus, in three frames, for a general motion, the null parallelepiped condition still holds i.e.,

N

N

0)( 113

112 NRNRN

Now, let we consider model the case B:

At time t1: (10)

At time t2 :

(11)

n p = N 0

)( 12121

120 n T R + NR = n p = N -

)( 13131

130 n T R + NR = n p = N -

At time t3:

(12)From equ. (11)

(13)

Applying to both sides of equ. (13) and notice that

n T R - N R = N --

121

121

12

N R-

112

0 = n N

T NT RN R -- 1212

112

112 = ) ( )(

113 13 130 ( )-N = n = N + np R R T

We get

(14)In the same way

(15)

Eliminate from eqs (14) and (15), we obtain:

(16)

If we define (17)

nT N = N R N - )( - 12

112

nT N N R N - )( - = 13

113

N RNT NN RNT N --

11312

11213 )( = )(

n

T R - R TH =

T R - R TG =

T R - R TF =

T)()(

T)()(

T)()(

T

T

T

123

133

1213

122

132

1213

121

131

1213

F, G, H are 3x3 matrices.

Then equ. (16) can be written in compact form:

(18)

Where

Note: equ. (18) is a vector equation containing 3 linear homogeneous equations. And only two of them are linear independent.

0 =

N H N

N G N

N F N

N

] [ = 312

212

11212 R | R | RR )()()(

] [ = 313

213

11313 R | R | RR )()()(

Therefore, 13 LC’s over 3 frames are needed to linearly solve for F, G, and H. Let we define:

We have

After E is found, translation can be solved by:

Subject to , let be

] [ = ] [ = 31212

21212

11212

3211212 RT | RT | RTE | E | E RTE = )()()()()()(

0 = E H

0 = EG

0 = E F

(3)

(2)

(1)

T 12

0 = 12121212 T R T = T ET

12 = T 12

T 12ˆ

Similarly, we define

Then

And

Subject to , let the solution be

R TE | E | EE )()()(1313

321 = ] [ =

0

0

0

3

2

1

= E H

= E G

= E F

)(T

)(T

)(T

0 = 13T E T

12 = T 13

T 13ˆ

In solving for R12 and R13, we rather reconstruct E and E’ for consistence (remember E and E’ were column by column).

are chosen such that

Rotations can then be solved by:

] ˆˆˆ [ ˆ

] ˆˆˆ [ˆ

121212132

131313121

T | H T | G TF T k = ET H | T G | T F T kE = TTT

2 = 2, = 22 EE

21,kk

0ˆ

0ˆ

1313

1212

= E - R T

- E = R T

Remark: check revered rotations:

if

if

Next, we determine relative amplitudes of translations

Let

substitute them into eqs (17):

R RR( 121212 , 1- = )det

R RR( 131313 , 1- = )det

T n = T ,T m = T 13131212 ˆˆ

= HT R - n R Tm

= GT R - n R Tm

= FT R - n R Tm

T)()(

T)()(

T)()(

T

T

T

ˆˆ

ˆˆ

ˆˆ

123

133

1213

122

132

1213

121

131

1213

when m, n are solved, translations are:

Build structures of 3D lines:Direction of 3D line :

Position of :

Choose sign to make

T n

m = T

T = T

1313

1212

ˆ

ˆ

l

) ||)( ||

)( +

|| )( ||

)(

2

11

13

113

112

112

N RN

N R N

N RN

N R( N = n

-

-

-

-

||n N||

n N )

||)N R( N||

N T + ||)N R||

( 2

11-

13

131-

12

( N

N T = Q 12

0 > z Q ˆ

l

3D Line

Check translation reflection. Evidently,

So for and

For and

n + Q = p : l

0 = N n Q = D

2

1313132

121212 )( )( + )( )( Nn RT + Q RN n RT + Q R = D+

T 12

T 13

T - 12

T - 13

2

131313

2

121212 )( )( + N )( )( Nn RT - Q Rn RT - Q R = D-

T T ,T T : D > D 13131212+

i

-

i

T - T ,T - T : D < D 13131212+

i

-

i

If

Else if

Motion from 2D Corners

First, the 3D structure of a corner is recovered easily from its image by introducing a new coordinate system;

Then, the rotation matrix R and translation vector T are computed from the recovered 3D corner correspondence;

Finally, it is concluded that one corner and two points correspondences over two views are sufficient to uniquely determine the motion.

1. Representations of 3D and image corners

3D corner:

Image corner:

][ 3210 nnnpc

][ 3210 NNNPC

),,()1,,(0

0

0

0

0

0

0

0000 z

z

z

y

z

x

z

pYXP

0p

2n

3n

1n

0),,( pnCBAN iT

iiii

is the normal of the projecting plane of the edge line:

iN

0 iii CYBXA

2. Recovering a 3D orthogonal corner from a single viewGiven the image corner of a 3D orthogonal corner:

to recover the 3D structure of the corner:

][ 3210 NNNPC

][ 3210 nnnpc

Introduce a new coordinate system

ii Nv

// , 0//pwi

, )//( iii wvu

such that

0 i i ip u v w

z

x

y

o

w

v uk

l

p0

P0 L

ii Nv

//

0// pwi

)//( iii wvu

Suppose that the edge line of a 3D corner has a slope in the coordinate

then the direction of the 3D edge line in the coordinate system is

ik

iii wvup 0

iii wvup 0

Tii kwvun ],0,1[),,(

w

S

xy

z

v

o

p

The axis of the coordinate has coordinate

in its own coordinate system However, in the , has the same direction with .

So,

iv

iii wvup 0

T)0,1,0(

iii wvup 0

xyzo iv

iN

iT NS

]0,1,0[

(1)

u

Similarly,

0

0

0 PP

Pwi

0PNwvu iiii

0]0,1,0[ PST

)(]0,0,1[ 0PNS iT

Write in a matrix form:

00 ||

100

010

001

PNPNS ii

0

0

i

i

N P

S N

P

0000

1 0

1

)||(),,( PkPN

k

PNPNnSwvunSn ii

i

iiiT

ii

From Eq.(1), the direction of the correspondent edge line in the xyzo

For orthogonal corners,

0,0,0 133221 nnnnnn

(2)

Substitute (2) into the above equations, and we can get three equations about the slope value

0

0

0

313

232

121

dkk

dkk

dkk 2311 / dddk

112 / kdk

133 / kdk

, 1, 2,3ik i

After and are found, the directions of the edge lines of the 3D corner can be easily computed

21,kk3k

000 ||||

1PkPN

Pn iii

Thus the 3D corner is reconstructed by

0 0 1 2 3| | | ,c z P n n n

)0( 0 z

Where is the 3D depth of the vertex of the corner, which cannot be determined from a single view.

0z

3. Determining the rotation matrix R

From the image corner correspondence,

To reconstruct

2'3

'2

'1

'0

13210

|||

|||

ttimeatNNNPC

ttimeatNNNPC

2'3

'2

'1

'00

132100

|||'

|||

ttimeatnnnPzc

ttimeatnnnPzc

The directions of the edge lines of the corner at time and are related by

1t 2t

)ˆ|ˆ|ˆ()ˆ|ˆ|ˆ( 321'3

'2

'1 nnnRnnn

TnnnnnnR )ˆ|ˆ|ˆ)(ˆ|ˆ|ˆ( 321'3

'2

'1

21 ˆ,ˆ nn 3n̂ are orthogonal unit vectors

Remark: since we get two sets of slope values , that means we recover two 3D corners from one image corner, so four rotation matrix can be computed. Therefore, additional information is needed to determine a unique solution.

ik

4. Determining the translation vector TSince the rotation matrix R has been computed, we can eliminate the rotation motion from the whole motion. Then there is only the motion of translation.

Intermediate time

Time t1

Time t2

TR

R

T

Following, we suppose there is only translation motion between time t1 and t2.

Remark: It is impossible to uniquely determine a translation from a single corner correspondence over two views of images. The rank of the coefficient matrix of the equations for translation is always less than 3.

Proof: A maximum of 4 equations can be derived from a single corner over two views of images: the three equations from edge lines, the other one from the vertex. Edge lines of the corner satisfy the equations :

'' '

0

|| || ( ) ( ), 1, 2, 3

|| ||i i

i i i i i

i i

N NN T sign N N P n N i

n N

(3)

Equations (3) and (4) are four linear equations about the unknown T, but they are not independent. The rank of the coefficient matrix is only 2. Another image point not lying in any of the three edge lines is needed to determine the translation.

YYYXXX '' ,

the other equation for the vertex of the corner is:

(4)

where

0)||( TYXXYXY

5. Getting a unique solution

Images over two frames

Four rotation matrix R

A corner correspondence

One translation T for each R

a corner and a nonsingular point correspondence

Unique solution R and T

Another nonsingular point correspondence

Uniqueness

If a 3D motion is a pure rotation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.

If a 3D motion is a pure translation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.

If a 3D motion is a rotation followed by a translation, an orthogonal corner correspondence and two point correspondences can uniquely determine the motion.

6. Motion estimation from a corner with known space angles

The process is the same as that in the orthogonal corner case. The only difference is :

0,0,0 133221 nnnnnn

Orthogonal corner

Corner with angles:

313232121 cosˆˆ,cosˆˆ,cosˆˆ nnnnnn

321 ,,

Image 1.1

Experiment Result:

Rotation: (0.170717, 0.973572, 0.151703; 33o54”)

Translation: (-0.227492 -0.956966 0.180176 )

Thanks