视觉的三维运动理解 刘允才 上海交通大学 2002 年 11 月 16 日 understanding 3d...
Post on 01-Jan-2016
503 Views
Preview:
TRANSCRIPT
视觉的三维运动理解刘允才
上海交通大学
2002 年 11 月 16 日
Understanding 3D Motion
from Images
Yuncai Liu
Shanghai Jiao Tong University
November 16, 2002
Understanding motions and structures of 3D scene
A basic problem of visual system and computer vision
3-D Motion
Non-rigid objectmotion
Articulated objectmotion
Rigid objectmotion
3D rigid motion understanding
Use 3D Features Use 2D Features
Monocular Images Binocular Images
Stereo Structured light
Points Lines Corners
Points Lines
Texture
Minimum Number of Correspondences for Motion Solution with Monocular Images
Correspondences Min Number Degeneration
3D-3D points 3 collinear
3D-3D lines 2 parallel
2D-3D points 3
2D-3D lines 3
2D-2D points 5 quadratic surface
2D-2D lines 6 (3 frames)
2D corners 1 CC + 2 PC
xo
y
o X
YP (X, Y)
P (x, y, z)
L
l
N
P0
focus length = f
r
R
Perspective Projection
z
yfY
z
xfX ,
p
Point:
pz
P 1
Line: AX + BY + C = 0
( , , )N A B C
N p l
is normal of projection plan !
3D motion expression
Motion of a rigid object in 3D is usually expressed as a rotation around system origin followed by a translation.
Let
z
y
x
p
be a 3D point of an object at time t1
z
y
x
p
be a 3D point of an object at time t2
R be rotation , a 3x3 orthonormal matrix.
z
y
x
T
T
T
T
be translation vector.
Then, from time t1 to t2 :
TpRp
)3()2()1(
333231
232221
131211
|| rrr
rrr
rrr
rrr
R
kiirrr kii ,3,2,1,0,1 )()(2)(
Note:
There are only 3 independent parameters in R.
Properties of rotation matrix
11 RRRRorRR TTT
UUR
VUVRUR
)( VURVRUR
R is expressed by rotation axis and rotation angle.
Let rotation axis be
where: n12+n2
2+n32=1.
Rotation is made by a rotating with an angle around a rotation axis.
3
2
1
ˆ
n
n
n
n
sin)cos1( 32121 nnnr
cos)1( 22
2222 nnr
sin)cos1( 13223 nnnr sin)cos1( 23131 nnnr sin)cos1( 13232 nnnr
cos)1( 23
2333 nnr
sin)cos1( 23113 nnnr sin)cos1( 32112 nnnr
cos)1( 21
2111 nnr
Then the elements of R are:
Express R by 3 rotation angles.
CCr 11
SCCSSr 12
SSCSCr 13
SCr 21
CCSSSr 22
κωκφω CSSSCr 23
Sr 31
CSr 32
CCr 33
Quaternion form of rotation:
Quaternion is a four-element vector,
which can be used to express a rotation:
Let a rotation around axis by angle
;,,, 3210 qqqqq 123
22
21
20 qqqq
2
22
12
32
010322031
10322
32
12
22
03021
203130212
32
22
12
0
)(2)(2
)(2)(2
)(2)(2
)(
qqqqqqqqqqqq
qqqqqqqqqqqq
qqqqqqqqqqqq
qR
3
2
1
ˆ
n
n
n
n
Quaternion product:
A 3D vector can be expressed as a quaternion with scalar part being zero:
Pure rotation in 3D:
Express the rotation by quaternion:
uuuququuqqqq 0000 ,
pkzjyixr
00
kzjyixp
pRp
qrqr
Motion from 2D PC
------- From 2D images to determine 3D motion.
------- At least 5-point correspondences over two- image view are required.
------- 3D translation can only be determined over a scale factor.
------- Degeneration case: 3D points are on a quadratic surface.
Assume:
------ Single stationary camera.
------ Central projection model.
------ Rigid moving object.
------ Focus length f = 1, thus
; z
y = Y ;
z
x = X
p z
1 = )1 Y, (X, = P T
Let 3D motion from of time to of
(1)
Where
From equ (1)
(2)Apply to both sides of equ (2)
(3)
p p
T + R
p = p
2t1t
Tzyx TTTT
rrr
rrr
rrr
R
;
,333231
232221
131211
T + R
Pz = P z T
P Tz = P T z
R
Apply to both sides of equ (3)
(4)Let we define
(5)The eq(4) can be rewritten as
(6)
Note: eq.(6) is a homogeneous scalar equation. is a matrix containing only motion parameters, 8 or more PCs can uniquely determine E, subject to:
P
0 = P R
T P
] R T | R T | R T [ = RT = E (3)(2)(1)
0 = P E
P
RT = E
33
22 E
R
pT
Rp
p’
PP’
RP
o 0 PRTP
After matrix E is found ,translation can be solved:
i.e.
(7)
T
0 = T R T = T E
0 = T ET
can be determined from eq(7) subject to T
12 T
Once is obtained, rotation R can be obtained by least-square method:
(8)
Or let
T
0 R - E = T
][
][321
321
R | R | R R =
E | E | E E = )()()(
)()()(
][ = 213132321 EE + TE | EE + TE | EE + TER )()()()()()()()()(
Note, 180o reflection of motion is still a solution of equ (7) (homogeneous equation). In this case, object is moving behind the camera. To check for a real solution, we apply to both sides of equ (2).
Therefore if z > 0, it must hold that
Thus if,
let
P
0 = T P + P R Pz
0 > )P T( )P R
P(
0 < )P T( )P R
P(i
T - T
Motion from LC’s
------ from two image frames of a single camera,
3D motion can never be solved.
------ over 3 frames, at least 6 LC’s are required.
------ motion models
------ for a linear algorithm, 13 LC’s are needed.
n R = n
n R = nT + p R = pT + p R = p
23
12
2323
1212
Model A
Model B
Relation between model A and B:
n R = n
n R = nT + p R = pT + p R = p
13
12
1313
1212
T + T R = T , R R = R
23122313231213
xo
y
o X
Y
L
l
P0
focus length = f
N
N’ R-1
L’
l’
For a pure rotation,
and
are collinear.
In case of three frames, three collinear vectors collinear, forming a null parallelepiped, i.e.,
NR
1 N
0)( 113
112 NRNRN
o
o
N’
L’
l l’
N
For a pure translation,
and lie in a plan that pass though origin and perpendicular to 3D line l .
Thus, in three frames, for a general motion, the null parallelepiped condition still holds i.e.,
N
N
0)( 113
112 NRNRN
Now, let we consider model the case B:
At time t1: (10)
At time t2 :
(11)
n p = N 0
)( 12121
120 n T R + NR = n p = N -
)( 13131
130 n T R + NR = n p = N -
At time t3:
(12)From equ. (11)
(13)
Applying to both sides of equ. (13) and notice that
n T R - N R = N --
121
121
12
N R-
112
0 = n N
T NT RN R -- 1212
112
112 = ) ( )(
113 13 130 ( )-N = n = N + np R R T
We get
(14)In the same way
(15)
Eliminate from eqs (14) and (15), we obtain:
(16)
If we define (17)
nT N = N R N - )( - 12
112
nT N N R N - )( - = 13
113
N RNT NN RNT N --
11312
11213 )( = )(
n
T R - R TH =
T R - R TG =
T R - R TF =
T)()(
T)()(
T)()(
T
T
T
123
133
1213
122
132
1213
121
131
1213
F, G, H are 3x3 matrices.
Then equ. (16) can be written in compact form:
(18)
Where
Note: equ. (18) is a vector equation containing 3 linear homogeneous equations. And only two of them are linear independent.
0 =
N H N
N G N
N F N
N
] [ = 312
212
11212 R | R | RR )()()(
] [ = 313
213
11313 R | R | RR )()()(
Therefore, 13 LC’s over 3 frames are needed to linearly solve for F, G, and H. Let we define:
We have
After E is found, translation can be solved by:
Subject to , let be
] [ = ] [ = 31212
21212
11212
3211212 RT | RT | RTE | E | E RTE = )()()()()()(
0 = E H
0 = EG
0 = E F
(3)
(2)
(1)
T 12
0 = 12121212 T R T = T ET
12 = T 12
T 12ˆ
Similarly, we define
Then
And
Subject to , let the solution be
R TE | E | EE )()()(1313
321 = ] [ =
0
0
0
3
2
1
= E H
= E G
= E F
)(T
)(T
)(T
0 = 13T E T
12 = T 13
T 13ˆ
In solving for R12 and R13, we rather reconstruct E and E’ for consistence (remember E and E’ were column by column).
are chosen such that
Rotations can then be solved by:
] ˆˆˆ [ ˆ
] ˆˆˆ [ˆ
121212132
131313121
T | H T | G TF T k = ET H | T G | T F T kE = TTT
2 = 2, = 22 EE
21,kk
0ˆ
0ˆ
1313
1212
= E - R T
- E = R T
Remark: check revered rotations:
if
if
Next, we determine relative amplitudes of translations
Let
substitute them into eqs (17):
R RR( 121212 , 1- = )det
R RR( 131313 , 1- = )det
T n = T ,T m = T 13131212 ˆˆ
= HT R - n R Tm
= GT R - n R Tm
= FT R - n R Tm
T)()(
T)()(
T)()(
T
T
T
ˆˆ
ˆˆ
ˆˆ
123
133
1213
122
132
1213
121
131
1213
when m, n are solved, translations are:
Build structures of 3D lines:Direction of 3D line :
Position of :
Choose sign to make
T n
m = T
T = T
1313
1212
ˆ
ˆ
l
) ||)( ||
)( +
|| )( ||
)(
2
11
13
113
112
112
N RN
N R N
N RN
N R( N = n
-
-
-
-
||n N||
n N )
||)N R( N||
N T + ||)N R||
( 2
11-
13
131-
12
( N
N T = Q 12
0 > z Q ˆ
l
3D Line
Check translation reflection. Evidently,
So for and
For and
n + Q = p : l
0 = N n Q = D
2
1313132
121212 )( )( + )( )( Nn RT + Q RN n RT + Q R = D+
T 12
T 13
T - 12
T - 13
2
131313
2
121212 )( )( + N )( )( Nn RT - Q Rn RT - Q R = D-
T T ,T T : D > D 13131212+
i
-
i
T - T ,T - T : D < D 13131212+
i
-
i
If
Else if
Motion from 2D Corners
First, the 3D structure of a corner is recovered easily from its image by introducing a new coordinate system;
Then, the rotation matrix R and translation vector T are computed from the recovered 3D corner correspondence;
Finally, it is concluded that one corner and two points correspondences over two views are sufficient to uniquely determine the motion.
1. Representations of 3D and image corners
3D corner:
Image corner:
][ 3210 nnnpc
][ 3210 NNNPC
),,()1,,(0
0
0
0
0
0
0
0000 z
z
z
y
z
x
z
pYXP
0p
2n
3n
1n
0),,( pnCBAN iT
iiii
is the normal of the projecting plane of the edge line:
iN
0 iii CYBXA
2. Recovering a 3D orthogonal corner from a single viewGiven the image corner of a 3D orthogonal corner:
to recover the 3D structure of the corner:
][ 3210 NNNPC
][ 3210 nnnpc
Introduce a new coordinate system
ii Nv
// , 0//pwi
, )//( iii wvu
such that
0 i i ip u v w
z
x
y
o
w
v uk
l
p0
P0 L
ii Nv
//
0// pwi
)//( iii wvu
Suppose that the edge line of a 3D corner has a slope in the coordinate
then the direction of the 3D edge line in the coordinate system is
ik
iii wvup 0
iii wvup 0
Tii kwvun ],0,1[),,(
w
S
xy
z
v
o
p
The axis of the coordinate has coordinate
in its own coordinate system However, in the , has the same direction with .
So,
iv
iii wvup 0
T)0,1,0(
iii wvup 0
xyzo iv
iN
iT NS
]0,1,0[
(1)
u
Similarly,
0
0
0 PP
Pwi
0PNwvu iiii
0]0,1,0[ PST
)(]0,0,1[ 0PNS iT
Write in a matrix form:
00 ||
100
010
001
PNPNS ii
0
0
i
i
N P
S N
P
0000
1 0
1
)||(),,( PkPN
k
PNPNnSwvunSn ii
i
iiiT
ii
From Eq.(1), the direction of the correspondent edge line in the xyzo
For orthogonal corners,
0,0,0 133221 nnnnnn
(2)
Substitute (2) into the above equations, and we can get three equations about the slope value
0
0
0
313
232
121
dkk
dkk
dkk 2311 / dddk
112 / kdk
133 / kdk
, 1, 2,3ik i
After and are found, the directions of the edge lines of the 3D corner can be easily computed
21,kk3k
000 ||||
1PkPN
Pn iii
Thus the 3D corner is reconstructed by
0 0 1 2 3| | | ,c z P n n n
)0( 0 z
Where is the 3D depth of the vertex of the corner, which cannot be determined from a single view.
0z
3. Determining the rotation matrix R
From the image corner correspondence,
To reconstruct
2'3
'2
'1
'0
13210
|||
|||
ttimeatNNNPC
ttimeatNNNPC
2'3
'2
'1
'00
132100
|||'
|||
ttimeatnnnPzc
ttimeatnnnPzc
The directions of the edge lines of the corner at time and are related by
1t 2t
)ˆ|ˆ|ˆ()ˆ|ˆ|ˆ( 321'3
'2
'1 nnnRnnn
TnnnnnnR )ˆ|ˆ|ˆ)(ˆ|ˆ|ˆ( 321'3
'2
'1
21 ˆ,ˆ nn 3n̂ are orthogonal unit vectors
Remark: since we get two sets of slope values , that means we recover two 3D corners from one image corner, so four rotation matrix can be computed. Therefore, additional information is needed to determine a unique solution.
ik
4. Determining the translation vector TSince the rotation matrix R has been computed, we can eliminate the rotation motion from the whole motion. Then there is only the motion of translation.
Intermediate time
Time t1
Time t2
TR
R
T
Following, we suppose there is only translation motion between time t1 and t2.
Remark: It is impossible to uniquely determine a translation from a single corner correspondence over two views of images. The rank of the coefficient matrix of the equations for translation is always less than 3.
Proof: A maximum of 4 equations can be derived from a single corner over two views of images: the three equations from edge lines, the other one from the vertex. Edge lines of the corner satisfy the equations :
'' '
0
|| || ( ) ( ), 1, 2, 3
|| ||i i
i i i i i
i i
N NN T sign N N P n N i
n N
(3)
Equations (3) and (4) are four linear equations about the unknown T, but they are not independent. The rank of the coefficient matrix is only 2. Another image point not lying in any of the three edge lines is needed to determine the translation.
YYYXXX '' ,
the other equation for the vertex of the corner is:
(4)
where
0)||( TYXXYXY
5. Getting a unique solution
Images over two frames
Four rotation matrix R
A corner correspondence
One translation T for each R
a corner and a nonsingular point correspondence
Unique solution R and T
Another nonsingular point correspondence
Uniqueness
If a 3D motion is a pure rotation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.
If a 3D motion is a pure translation, an orthogonal corner correspondence and a nonsingular point correspondence over two frames can uniquely determine the motion.
If a 3D motion is a rotation followed by a translation, an orthogonal corner correspondence and two point correspondences can uniquely determine the motion.
6. Motion estimation from a corner with known space angles
The process is the same as that in the orthogonal corner case. The only difference is :
0,0,0 133221 nnnnnn
Orthogonal corner
Corner with angles:
313232121 cosˆˆ,cosˆˆ,cosˆˆ nnnnnn
321 ,,
Image 1.1
Experiment Result:
Rotation: (0.170717, 0.973572, 0.151703; 33o54”)
Translation: (-0.227492 -0.956966 0.180176 )
Thanks
top related