robot vision 15678
TRANSCRIPT
ROBOT VISION
Lecture Notes
로봇 비젼
강의 노트
Course No. 15678
Credits Hours 3
Instructor:
Professor
Choi, Tae-Sun
Course outline:
The principles of the machine/robot vision are introduced. It covers image formation,
pattern classification, motion and optical effect for object recognition. In addition, the
design technology of the robot vision system with optical device is studied.
Prerequisite: Digital Signal Processing, Image Processing.
Textbook and References: 1. Robot Vision, B. K. P. Horn, MIT Press.
2. Computer Vision, Dana Ballard and Christopher
Brown, Prentice Hall.
Course Schedule
1. Image Formation & Image Sensing
2. Binary Images: Geometrical Properties
3. Binary Images: Topological Properties
4. Regions & Image Segmentation
5. Image Processing: Continuous Images
6. Image Processing: Discrete Images
7. Edge & Edge Finding
8. Lightness & Color
9. Reflectance Map: Photometric Stereo
10. Reflectance: Shape from Shading
11. Motion Field & Optical Flow
12. Photogrammetry & Stereo
13. Pattern Classification
14. Polyhedral Objects
15. Extended Gaussian Images
16. Passive Navigation & Structure from Motion
1
1. Introduction
I. Human vision system
Figure 1—1
• Rod : more sensitive to light
Scotopic vision
Slender receptor
• Cone : shorter & thicker in structure
Photopic vision : R.G.B
6.5 million
• Fovea : Density of cones is greatest
The region of sharpest photopic vision Camera visual system
2
Figure 1—2
i. Photometric information
brightness : (ex: bright green, dark green)
color (ex, R. G. B)
ii. Geometric information
shape
Figure 1—3
2
NTSC : US standard for TV picture and audio coding and transmission
(timing calculation)
For one frame : 512×512 = 256k Byte
For RGB : 3
For real time : 30 frames/sec
256k × 3 × 30 = 22.5M Byte/sec
180 M bit/sec
180 Mbps T1 1.544 Mbps
Compression ratio : 180
1201.544
≈
Wavelet compression
MPEG I, II, IV
Compression JPEG
Fractal
H.261
II. Robot vision
RV
Computer graphics
Image processing
compressio
3
Figure 1—4
Figure 1—5
4
“Scene description”
Under constrained problems : Inverse problem
for example 2 equations
3 unknown variables are given
1. Image analysis :a detailed but undigested description
2. Scene analysis : more parsimonious, structured descriptions suitable for
decision marking
The apparent brightness of a surface depends on three factors
1. Microstructure
2. Distribution of incident light
3. Orientation of the surface w. r. t light source & observer
5
Figure 1—6
“Lambertian Surface”
One that appears equally bright from all viewing directions and reflects all incident light,
absorbing none
6
2. Image Formation & Image sensing
What determines where(Shape-Geometric) the image of some point will appear?
What determines how(Brightness, Color – Photonetiz) bright the image of some surface
will be?
I. Two Aspects of Image Formation
i. Perspective projection
Figure 2—1
7
ii. Orthographic Projection
2 2
2 20
' '' (magnification)
x yfm
z x y
δ δ
δ δ
+= =
+
In case the depth range of a scene is small relative to the distance z0
Figure 2—2
⇒
=
=
myy
mxx
'
'
if m=1
=
=
yy
xx
'
'
=> This orthographic projection can be modeled by rays parallel to the optical axis
(rather than one passing through the origin)
(a) (b)
Figure 2—3
I
pE
δ
δ= [watt/m]
Where δP is the power of the radiant energy falling on the
8
II. Brightness
1. Image Brightness : Irradiance 2 [ / ]
PE watt m
I
δ
δ=
2. Scene Brightness :
L=2
2 [ / ]
PL watt m
I W
δ
δ δ= where δ2
P is the power emitted by the infinitesimal surface patch
of area (δI) into an infinitesimal solid angle (δw)
2, where is area, and is Distance
ASolidAngle A D
D≡
Ex 1 Hemisphere
wδ
R
2
2
4 1Solid angle = = 2
2
Whole sphere = 4
R
R
ππ
π
⋅
Figure 2—4
Ex 2 Small patch
9
Figure 2—5
Lens
Figure 2—6
cos
( / cos )2
Image Irradiance
PE
I
ISolid angle
f
δ
δδ α
α
=
=
0
2
= '
cos
( / cos )
Scene Radiance
PL
O w
Solid anglez
δ
δ δδ θ
α=
Transfer Function: ( )L
fE
δ
δ=i
10
f(·) : BRDF(Bidirectional Reflectance Distribution Function) in Ch 10
Ex3) Sphere/Radius 2m
Figure 2—7
III. Lenses
Figure 2—8
i. Lens formula
1 1 1
'f z z= + 2–1
' '
2
s z z
R D
−= 2–2
Solve (2-1) & (2-2) to calculate R – blur circle radius
11
1 1 1( )
2
DSR
f s z= − −
R = 0 at focused point P′
ii. Depth of field (DOF)
The range of distances over which objects are focused “sufficiently well” in the sense
that thediameter of the blur circle is less than the resolution of the imaging device.
The larger the lens aperture => the less the DOF
Figure 2—9
iii. Vignetting
In a simple lens, all the rays that enter the front surface of the lens end up being focused in the image
Figure 2—10
12
In a compound lens, some of the rays that pass through the first lens may be occluded
by portions of the second lens, and so on … Vignetting is a reduction in light-gathering
power with increasing inclination of light rays with respect to the optical axis
Figure 2—11
iv. Aberration
Points on the optical axis may be quite well focused, which those in a corner of the
image are smeared out
13
IV. Image sensing
i. Color sensing
a) Human eye
Figure 2—12
b) CCD sense
R, G, B filter
R(λ) : 1 ( at Red color)
14
0 (otherwise)
G(λ) : 1 (at Green color)
0 (otherwise)
B(λ) : 1 (at Blue color)
0 (otherwise)
→ UV filter filtered out UV ray
ii. Randomness and Noise
a) Random variable [R.V]
Consider an experiment H with sample description space Ω the elements on points of
Ω,δ, are the random out comes of H
If to every δ, we assign a real number X(δ) The real establish a correspondence rule
between δ and R the real time, such a rule subject to certain constraints is called a
random variable
15
Figure 2—13
R.V.
Probability density function (p.d.f) p(x)
Where p(x) ≥ 0 for all x, therefore: ( ) 1p x dx
∞
−∞
=∫
mean :
( )xp x dxµ∞
−∞
= ∫ : first moment
variance :
2 2 2( ) ( ) ( ) : second momentx p x dx x p x dxδ µ∞ ∞
−∞ −∞
= − =∫ ∫
Cumulative probability distribution (CDF) P(x)
( ) ( )
x
P x p x dt−∞
= ∫
Two R.V.
SVR
XXX
..
21 +=
P(X1) & P(X2) what P(x) = ?
Solution
16
Figure 2—14
Given x2 , x1 : x-x2
1 1 1 1 2( ) ( )P x x P x x xδ δ⋅ = −
Now, x2 can take on a range of values P2(x2).δ(x2)
To find the prob. that x lies between x and x+δx,
1 2 2 2 2
1 2 2 2 2
1 2
1 2
( ) ( ) ( )
( ) ( ) ( )
( ) ( )
P x x P x x xP x dx
P x P x x P x dx
P x t P t dt
P P
δ δ∞
−∞
∞
−∞
∞
−∞
⋅ = −
∴ = −
= −
= ∗
∫
∫
∫ ∫
For multiple RV's : ∑=
=N
i
ixN
x1
1
Mean value : µ
µ=
N
N
Variance : 2 2
2
N
N N
σ σ=
Standard Deviation : N
σ⇒
b) Gaussian (Normal)
p.d.f :
21
21( )
2
x
p x e
µ
σ
πσ
− −
=
17
c) Poisson (m>0)
p.d.f : ( )!
n
m mp k e
n
−= ⋅
where n arrivals in a time interval T for some m
mean = m variance = σ2
Ex Let X and Y be independent r.v.'s with ( ) ( )x
xp x e u x
−= and
1( ) [ ( 1) ( 1)]
2y
p y u y u y= + − − and Let Z X Y≡ + , what is the p.d.f of Z?
(a) (b)
Figure 2—15
18
( )
1
( ) ( )* ( ) ( ) ( )
(a) 1 ; ( ) 0
1(b) 1 1 ; ( )
2
(c) z 1 ; (
z x y x y
z
z
z y
z
z
p z p x p y p z y p y dy
z p z
z p z e dy
p z
∞
−∞
− −
−
= = −
< − =
− ≤ < =
≥
∫
∫1
( )
1
1)
2
z ye dy− −
−
= ∫
Figure 2—16
19
3. Binary images : Geometric properties
I. Binary Images
Figure 3—1
1. Area of the object 1 sin cos 0
sin cos
x yx yθ θ ρ
ρ ρ
θ θ
+ = ⇒ − + =
−
( , )I
A b x y dxdy= ∫ ∫ ; zero-th Moment
2. Position of the object : center of area
1( , )x xb x y dxdy
A= ∫ ∫ : First moment
1( , )y b x y dxdy
A= ∫ ∫
3. Orientation of the object
(ρ,θ)
2 ( , )I
E r b x y dxdy= ∫ ∫ : second moment
20
II. Simple Geometrical Properties
Figure 3—2
2 2 2
0 0
2 2
2 2 2 2
( ) ( )
( ( sin cos )) ( ( cos sin ))
( ) 2 ( sin cos ) 2 ( cos sin )
r x x y y
x s y s
x y x y s x y s
ρ θ θ ρ θ θ
ρ ρ θ θ θ θ
= − + −
= − − + + − +
= + + + − − + +
To find the shortest distance from (x, y) to (x0, y0) differentiate with respect to S and set
equal to zero
2( )
0d r
ds=
θθ sincos yxS +=
2 2
0
2
(sin cos ) sin ( cos sin )cos
sin sin sin cos
sin (sin cos )
x x x x y
x y
y
θ θ ρ θ θ θ θ
θ ρ θ θ θ
θ θ θ ρ
− = + + − +
= + −
= − +
)cossin(cos0 ρθθθ +−−=− yxyy
2
0
2
0
2 )()( yyxxr −+−=∴
2)cossin( ρθθ +−= yx
21
If, r=0 , ;0cossin =+− ρθθ yx
;0cossin),,( =+− ρθθ yxyxAt
Finally, 2 ( , )E r b x y dxdy= ∫ ∫
2( sin cos ) ( , )I
x y b x y dxdyθ θ ρ= − +∫ ∫
Differentiating w.r.t ρ and setting the result to zero lead to
2( sin cos ) ( , ) 0I
dEx y b x y dxdy
dθ θ ρ
ρ= − + =∫ ∫
0)cossin( =+− ρθθ yxA
Let '' xxxxxx −=⇒−=
yyy −='
θθρθθ
ρθθ
cos'sin'cossin
0)cos)'(sin)'(
yxyx
yyxx
−=+−
=+−−−
and so
2
2 2
( 'sin 'cos ) ( , )
' sin ( , ) ( 2 ' ') cos sin
I
I I
x y b x y dxdy
x b x y dxdy x y
θ θ
θ θ θ
−
= + −
∫ ∫
∫ ∫ ∫ ∫
2 2
2 2
( , ) ' cos ( , )
sin sin cos cos
Ib x y dxdy y b x y dxdy
a b c
θ
θ θ θ θ
+
= − +
∫ ∫
where,
22
2
2
( ') ( , ) ' '
2 ( ' ') ( , ) ' '
( ') ( , ) ' '
I
I
I
a x b x y dx dy
b x y b x y dx dy
c y b x y dx dy
=
= ⋅
=
∫ ∫
∫ ∫
∫ ∫
Now
θθ 2sin2
12cos)(
2
1)(
2
1bcacaE −−−+=
Differentiating w.r.t θ and setting the result to zero, we have
ca
b
−=θ2tan
unless 0 ; b a c= =
+−=
−=
∴
−
θθρ
θ
cossin
tan2
1 1
yx
ca
b
III. Projections
Figure 3—3
23
∫ ∫== dxyxbxVdyyxbxh ),()(,),()(
Area :
( , ) ( )
( )
A b x y dxdy h x dx
v y dy
= =
=
∫ ∫ ∫
∫
Position :
∫
∫ ∫ ∫
=
==
dyyyvA
y
dxxxhA
dxdyyxxbA
x
)(1
)(1
),(1
Orientation : ca
b
−= −1tan
2
1θ
( , )ρ θ θθρ cossin yx +−=
2 2( , ) ( )Ix b x y dxdy x h x dx=∫ ∫ ∫
2 2( , ) ( )I
y b x y dxdy y v x dy=∫ ∫ ∫
∫ ∫∫∫ ∫ −−= dyyhyxvxdttdtdxdyyxxybI )(2
1)(
2
1)(),( 222
Diagonal projection (θ=45°)
Figure 3—4
24
Suppose that θ is 45°
∫ ∫ +−== dsststbdxdyyxbtd )(2
1),(
2
1(),()(
Now consider that
2 2
2 2
1( ) ( , ) ( )
2
1 1( ) ( , )2 2
I I
I
x y b x y dxdy t d t dt
x xy y b x y dxdy
+ =
= + +
∫ ∫ ∫ ∫
∫ ∫
so, 2 2 21 1( , ) ( ) ( ) ( )
2 2Ixyb x y dxdy t d t dt x h x dx y v y dy= − −∫ ∫ ∫ ∫ ∫
IV. Discrete Binary Images
(reference of book figure 3-8)
area ∑∑= =
=N
i
M
j
ijbA1 1
position ),( ji
∑∑
∑∑
=
=
i j
ij
i j
ij
jbA
j
ibA
i
1
1
orientation ( , )ρ θ
∑∑
∑∑
∑∑
i j
ij
i j
ij
i j
ij
ijb
bj
bi
2
2
25
V. Run-Length coding
Figure 3—5
Where rik is the k-th run of the i-th line and the first run in each row is a run of zeros
2
.........., 6
1
2
1
4
1
22
i
ii
n
i
m
k
i
n
i
iki
mrrrrrA
i
+++==∑∑ ∑= = =
position (center of area) : ),( ji
26
∑∑
∑
==
=
=⋅=
=
n
i
j
n
i
i
k
m
k
ii
jvA
jhiA
i
rh
i
11
2
2
1
1..........
1
,
jv
1. Find the first horizontal differences of the image data
Figure 3—6
2. The first difference of the vertical projection can be computed from the
projection of the first
27
horizontal differences
⇒ count the number of circles subtracted by the number of triangles.
3. The vertical projection vj can be found by summing the result from left to right
Orientation ( , )ρ θ
j
ji
jt
ti j
ij
j
j
i
i
vjhidtijb
vj
hi
∑∑∑∑∑
∑
∑
−−= 222
2
2
2
1
2
1
Where, )(2
1jit +=
dt can be obtained in a way similar to that used to obtain the vertical projection
28
4. Binary Image : Topological properties
Jordan curve theorem
A simple closed curve separate the image into two simply connected regions
Figure 4—1
4 Connectedness – only edge-adjacent cells are considered neighbors.
Figure 4—2
29
Ex 1
Figure 4—3
4 objects
` 2 backgrounds
No closed curve ⇒ contradiction(by Jordan curve theorem)
8 connectedness – Corner-adjacent cells are considered neighbors, too.
Figure 4—4
6 connectedness
30
Figure 4—5
Figure 4—6
Figure 4—7
i. A sequential Labeling Algorithm
Recursive Labeling
1. Choose a point where bij=1 and assign a label to this point and into neighbors
31
Figure 4—8
2. Next, label all the neighbors of these neighbors ⇒ one component will have
been labeled completely
3. Find new places to start a labeling operation whenever an unlabeled point is
found where
bij=1
4. Try every cell in this scan
Sequential Labeling
32
Figure 4—9
1. if A is zero, there is nothing to do.
Figure 4—10
2. if A is one, if D has been labeled, simply copy that label and move on if not [if
one B orc is labeled, copy that label else, choose a new label for A]
(go to step 2)
33
II. Local counting and iterative modification
i. Local counting
Figure 4—11
Horizontal N
Vertical N Total Length = 2N
34
Figure 4—12
The overestimated rate : 1:22:2 =NN
Considering all slopes overestimated average ratio : 4
π
Overestimated average rate : π
4
-Euler number-
1. No. of Bodies – No. of Holes
2. No. of upstream convexities – No. of upstream concavities
convexities(+1) concavities(-1)
Figure 4—13
35
Figure 4—14
Body hole
1. body - hole : E = 1- 2 = -1 2 - 0 = 2 1 - 0 = 1 1 - 1 = 0 1 - 0 = 1 1 - 0 = 1
2. convexity - concavity : 1- 2 = -1 2 – 0 = 2 1 – 0 = 1 1 – 1 = 0 1 – 0 = 1 1 - 0 = 1
ii. The additive set property
Figure 4—15
36
⇒This permits to split an image up into smaller pieces and obtain an overall answer by
combining the
results of operations performed on the pieces
Figure 4—16
1 2 2 2 1
)()()()()()( eEdEcEbEaEedcbaE ++++=∪∪∪∪
[ ( ) ( ) ( ) ( )]E a b E b c E c d E d e− + + +∩ ∩ ∩ ∩
1-0 = 1 2-0 =2 3-0 =3 2-0 =2
(1 2 2 2 1) (1 2 3 2) = 0= + + + + − + + +
right (body = 1, hole 1) 1-1 =0
convexities =2
1(body) –
37
concavities =2
Ex1) hand
Figure 4—17
Body : 1 Hole : 0 1-0 =1
Convexities : 5 , Concavities : 4 5-4 =1
Euler number = 1
iii. Iterative modification
Instead of adding up the outputs of Local operators in the previous section, the new
binary image determined by the iterative method can be used as input to another cycle
of computation this process, called iterative modification is useful because it allows us
convexit
concavit
38
to incrementally changes an image that is difficult to process into one that might
succumb to the methods already discussed
Euler Differential E*
2. E* = - ( 1 – 1 = 0 ) + ( 1- 0 ) = 1
Part pattern Current pattern
2
3
39
4
5
**EEnew −=
40
5. Regions & image segmentation
I. 5.1 Thresholding Methods
•Image segmentation : An image segmentation is the partition of an image into a set of
non-overlapping regions whose union is the entire image
Figure 5—1
Detection of discontinuities : points, lines, and edges
Figure 5—2
41
Figure 5—3
∑=
=+⋅⋅⋅⋅⋅++=q
i
iiqq zwzwzwzwR1
2211
A mark used for detecting isolated points
Figure 5—4
i. Point detection
The detection of isolated points in an image is straightforward
If |R| > T when T is threshold, it’s an isolated point
ii. Line detection
The next level of complexity involves the detection of lines in an image. Consider
the four masks, as shown below
42
Figure 5—5
• If at a certain point in the image, |Ri|>|Rj| for all j≠i that point is said to be
more likely associated with a line in the direction of mask i, for example, if
|R1|>|Rj| , for j=2,3,4 the line is more likely associated with a horizontal line.
iii. Edge detection
The idea underlying most edge detection technologies is the computation of a local
derivative operator
Figure 5—6
43
Gradient operator
∂
∂
∂
∂
=∇
y
f
x
f
f
22
∂
∂+
∂
∂=∇
y
f
x
ff
Figure 5—7
Laplacian Operator
2
2
2
22
y
f
x
ff
∂
∂+
∂
∂=∇
)(4 86425
2zzzzzf +++−=∇ in discrete form
Figure 5—8
44
2
2
5 2 4 6 8
Cross Section of f
f = 4 (z )z z z z
∇
∇ − + + +
II. Thresholding
• Thresholding is one of the most important approaches to image segmentation
Figure 5—9
1 if ( , )( , )
0 otherwise
f x y Tg x y
<=
III. Region-oriented segmentation
i. Basic Formulation
Figure 5—10
45
Let R represent the entire image region and Ri the partitioned region for n=1,2,………n
(a) RRin
i
==
∪1
(entire image region)
(b) i
R is a connected region, i=1,2,……n
(c) i j
R R∩ = φ for all i & j , i ≠j
≠
=
=
φ
φ
φ
11
21
51
RR
RR
RR
∩
∩
∩
(d) )( iRP = True for i=1,2,…….n
Figure 5—11
ii. Region Growing by Pixel Aggregation
“Region Growing” is a procedure that groups pixels or sub-regions into larger regions.
The simplest of these approaches is pixel aggregation, which starts with a set of “seed”
points and from these grows regions by appending to each seed
Point those neighboring pixels that have similar properties
46
Ex 1
Figure 5—12
iii. Region splitting and Merging
Spitting
Let R represent the entire image region and select a predicate P as discussed in the
previous section For a square image, one approach for segmenting R is to subdivide it
successively into smaller and smaller quadrant regions so that, for any region Ri,
P(Ri) = TRUE that is, if P(Ri or R) = FALSE, divide the image into quadrants
Figure 5—13
Merging
If only splitting were used, the final partition likely would contain adjacent regions
With identical properties, this draw back may be remedied by allowing merging as well
as splitting two adjacent regions Rj &Rk are merged only if P(Rj∪Rk)=TRUE
Figure 5—14
47
• The preceding discussion may be summarized by the following procedure in
which, at any step, we split into four disjointed quadrants any region Ri where
P(Ri)= FALSE
Figure 5—15
a) merge any adjacent regions Rj & Rk for which P(Rj∪Rk) =TRUE ;
b) stop when no further merging and splitting is possible
Ex 2
Figure 5—16
MergingFALSERP
TRUERRRPFALSERP
FALSERPFALSERP
FALSERPTRUERP
TRUERPFALSERP
→+
++
=+
==
==
)(
)()(
)()(
)()(
)()(
4
4132233
432
341
21
∪∪
Figure 5—17
48
6. Image Processing : Continuous images
I. Linear, Shift-Invariant Systems
Spatial domain → Point Spread Function (PSF)
Frequency domain → Modulation Transfer Function (MTF)
Figure 6—1
• Defocused image ( g ) is a processed version of the focused image ( f )
Double the brightness of ideal image and the defocused image. If the image system
moves, f & g moves the same amount.
Figure 6—2
Linearity : 1 2 1 2( , ) ( , ) ( , ) ( , )f x y f x y g x y g x yα β α β+ → +
Shift : ( , ) ( , )f x a y b g x a y b− − → − −
49
II. Convolution
( , ) ( , ) ( , )f x y h x y g x y→ →
( , ) ( , ) ( , )g x y f x y h d d
g f h
ε η ε η ε η∞ ∞
−∞ −∞
= − −
→ = ⊗
∫ ∫
Linear ( ) ( )1( , ) 2 ( , ) ?f x y f h g x yα β+ → → =
[ ]1 2( , ) ( , ) ( , ) ( , )g x y f x y f x y h d dα ε η β ε η ε η ε η∞ ∞
−∞ −∞
= − − + − − ⋅∫ ∫
1 2( , ) ( , )g x y g x yα β= +
Shift invariant
( , ) ( , ) ( , )f x a y b h x y g x y− − → →
( , ) ( , ) ( , )g x y f x a y b h d dε η ε η∞ ∞
−∞ −∞
= − −∫ ∫
Impulse response
( , ) ( , ) ( , )f x y h x y h x y→ →
( , ) ( , ) ( , )g x y f x y h d dε η ε η ε η∞ ∞
−∞ −∞
= − −∫ ∫
and ( , ) ( , )g x y h x y=
Only possible if Origin
( , )0 otherwise
f x y∞
=
This function is called unit impulse ( , )x yδ or Dirac delta function
( , ) 1x y dxdyδ∞ ∞
−∞ −∞
=∫ ∫
50
Shifting Property : ( , ) ( , ) (0,0)x y h x y dxdy hδ∞ ∞
−∞ −∞
=∫ ∫
and ( , ) ( , ) ( , )x y h d d h x yδ ε η ε η ε η∞ ∞
−∞ −∞
− − ⋅ =∫ ∫
=> ( , )h x y : Impulse Response of the system
Figure 6—3
( , ) ( , )k x yε η δ ε η− − : scaled delta function
( , ) ( , ) ( , )f x y f x y d dε η δ ε η ε η∞ ∞
−∞ −∞
= ⋅ − −∫ ∫
( , ) ( , ) ( , )g x y f x y h x y d dε η ε η∞ ∞
−∞ −∞
= − − ⋅∫ ∫
b a⊗ = ( , ) ( , )a x y b x y d dε η ε η∞ ∞
−∞ −∞
− − ⋅∫ ∫ <= Let ,x ε α− = y η β− =
= ( , ) ( , )a b x y d d b aα β α β α β∞ ∞
−∞ −∞
⋅ − − = ⊗∫ ∫
Similarly, ( ) ( )a b c a b c⊗ ⊗ = ⊗ ⊗
51
Cascade
1 1 2( )1 2
f h f h hf h h
⊗ ⊗ ⊗→ → → => 1 2( )f h h⊗ ⊗
1 2f h h g→ ⊗ →
III. MTF
An Eigen function of a system is a function that is reproduced with at most a change in
amplitude.
( )jwt jwte A w e→ →
2 ( )( , ) i ux vyf x y e π +→ =
2 ( ) ( )( , ) ( , )
i u x v yg x y e h d d
π ε η ε η ε η∞ ∞
− + −
−∞ −∞
= ⋅∫ ∫
= 2 ( ) 2 ( ) ( , )i ux vy i u ve e h d dπ π ε η ε η ε η∞ ∞
+ − +
−∞ −∞
⋅∫ ∫
= 2 ( ) ( , )i ux vye A u vπ + ⋅
IV. Fourier Transform
Represent the signal as an infinite weighted sum of an infinite number of sinusoids (u: angular frequency)
( ) ( ) iuxF u f x e dx∞
−
−∞= ∫
Note:
cos sin 1ike k i k i= + = −
52
Arbitrary function => Single Analytic Expression
Spatial Domain(x)=> Frequency Domain(u) (Frequency Spectrum F(u))
Inverse Fourier Transform (IFT)
( ) ( )1
2
iuxf x F u e du
π
∞
−∞= ∫
( )
2
1( , ) ( , )
4
i ux vyf x y F u v e dudvπ
∞ ∞+ +
−∞ −∞
= ∫ ∫
V. The Fourier Transform of Convolution
Let g f h= ⊗
Now,
( ) ( )
( ) ( )
( ) ( ) ( )
( ) ( )
( ) ( )
2
2
22
2 2 '
' '
i ux
i ux
i u xi u
i u i ux
G u g x e dx
f h x e d dx
f e d h x e dx
f e d h x e dx
F u H u
π
π
π τπ τ
π τ π
τ τ τ
τ τ τ
τ τ
∞ ∞−
−∞ −∞
∞ ∞−
−∞ −∞
∞ ∞− −−
−∞ −∞
∞ ∞− −
−∞ −∞
=
= −
= −
=
=
∫ ∫
∫ ∫
∫ ∫
∫ ∫
Convolution in spatial domain Multiplication in frequency domain
Spatial Domain (x) Frequency Domain(u)
g f h G FH
g fh G F H
= ⊗ ↔ =
= ↔ = ⊗
53
So, we can find g(x) by Fourier transform
Figure 6—4
Spatial Domain (x) Frequency Domain (u)
Linearity ( ) ( )1 2c f x c g x+ ( ) ( )1 2c F u c G u+
Scaling ( )f ax 1 uF
a a
Shifting ( )0f x x− ( )02i uxe F u
π−
Symmetry ( )F x ( )f u−
Conjugation ( )f x∗ ( )F u
∗ −
Convolution ( ) ( )f x g x⊗ ( ) ( )F u G u
Differentiation ( )n
n
d f x
dx
( ) ( )2n
i u F uπ
54
VI. Generalized Functions and Unit Impulses
• The integral of the one-dimensional unit impulse is the unit step function,
( ) ( )x
t dt u xδ−∞
=∫ , where ( )u x = 1, for 0;x ≻
1
2= , for 0;x =
0= , for 0x ≺ .
What is the Fourier transform of the unit impulse?
We have ( )( , ) 1i ux vyx y e dxdyδ∞ ∞
− +
−∞ −∞
=∫ ∫ , as can be seen by substituting 0x = and 0y =
into ( )i ux vye− + , using the sifting property of the unit impulse. Alternatively, we can use
( )
0lim ( , ) i ux vyx y e dxdyεε
δ∞ ∞
− +
→−∞ −∞
∫ ∫ , or 0
1 1lim
2 2
iux ivye dx e dy
ε ε
εε εε ε
− −
→− −
∫ ∫ ,
That is 0
sin sinlim 1
u v
u vε
ε ε
ε ε→= .
VII. Convergence Factors and the Unit Impulse
We want a smoothed function of f(x)
( ) ( ) ( )g x f x h x= ∗
Let us use a Gaussian kernel
Figure 6—5
55
( )2
2
1 1exp
22
xh x
σπσ
= −
Then
( ) ( )2 21
exp 22
H u uπ σ
= −
( ) ( ) ( )G u F u H u=
H(u) attenuates high frequencies in F(u) (Low-pass Filter)!
VIII. Partial Derivatives
What is the F.T. of ( , )x yf
x
∂
∂and
f
x
∂
∂ ?
2 ( )i ux vyfe dxdy
x
π∞ ∞
− +
−∞ −∞
∂
∂∫ ∫ = 2 2ux vyfe dx e dy
x
π π∞ ∞
− −
−∞ −∞
∂⋅
∂ ∫ ∫
2 uxfe dx
x
π∞
−
−∞
∂
∂∫= 2 2( , ) ( 2 ) ( , )iux iux
f x y e iu f x y e dxπ ππ
∞∞− −
−∞−∞
− − ⋅∫
= 2 2( , ) (2 ) ( , )iux iuxf x y e iu f x y e dx
π ππ∞
∞− −
−∞−∞
+ ⋅∫
We can’t proceed unless, lim ( , ) 0x
f x y→±∞
=
In that case,
2 ( )2 ( , ) i ux vyiu f x y e dxdy
ππ∞ ∞
− +
−∞ −∞
∫ ∫ =(2 ) ( , )iu F u vπ ⋅
( , )
( , )
2 ( , )
2 ( , )
x y
x y
fF iu F u v
x
fF iv F u v
x
π
π
∂ ∴ = ⋅
∂
∂ = ⋅
∂
56
IX. Rotational Symmetry and Isotropic operators
• Rotationally symmetric operators are particularly attractive because the treat
image features in the same way, no matter what their orientation is.
• Circular symmetry
Figure 6—6
Let us introduce polar coordinates in both spatial and frequency domain
Figure 6—7
cosx r φ= and siny r φ= cos( )ux vy r ρ φ α+ = ⋅ −
cosu ρ α= ⋅ and sinv ρ α=
57
2 ( )( , ) ( , ) i ux vyF f x y f x y e dxdy
π∞ ∞
− +
−∞ −∞
= ∫ ∫
2 cos( )
0 0
( ) i rrf r e rdrd
π φ α φ∞ ∞
− ⋅ −= ⋅∫ ∫
Zeroth order Vessel function
(Hankel Transform) => 0( ) 2 ( ) (2 )rF f r J r rdrρ ρ π π ρ∞
−∞
= ∫
0
0
( ) 2 ( ) (2 )fr r f J r dρπ ρ π ρ ρ ρ∞
= ∫
where 2
cos( )0
0
1( )
2
i x wJ x e d
πθ θ
π− ⋅ −= ∫
X. Blurring
2 2
2
1
2
2
1( , )
2
x y
h x y e σ
πσ
+−
=
. .F T⇓
2 2
2
1
2 ( )2
2
1( , )
2
x y
i ux vyH u v e e dxdy
πσ
πσ
+∞ ∞−
− +
−∞ −∞
= ⋅∫ ∫
2 21 1
2 22 21 1
2 2
x y
iux ivye e dx e e dy
π πσ σ
πσ πσ
∞ ∞− − − −
−∞ −∞
= ⋅ ⋅∫ ∫
21
221
2
x
iuxe e dx
πσ
πσ
∞− −
−∞
∫
2 21 1
2 2cos(2 ) sin(2 )
x x
e ux dx i e ux dxσ σπ π ∞ ∞
− −
−∞ −∞
= − ⋅ ⋅∫ ∫
58
The second integral is odd so, integral over symmetric region is zero.
And,
2 2 21
2
2
1cos(2 ) ,
2
x u
ae ux dx e aa
πσ π
πσ
∞− −
−∞
= ⋅ =∫
2
2 2 21 1
(2 )2 2cos(2 ) 2
xu
e ux dx eπ σ
σ π π σ ∞
− − ⋅ ⋅
−∞
= ⋅ ⋅∫
2 2 2 2 2 21 1(2 ) (2 )
2 21 1
( , ) 2 22 2
u v
H u v e eπ σ π σ
πσ πσπ σ π σ
− ⋅ ⋅ − ⋅ ⋅
= ⋅ ⋅ ⋅ ⋅ ⋅⋅ ⋅
2 2 2 21
(2 ) ( )2
u v
eπ σ− ⋅ + ⋅
=
Which is also rotationally symmetric. F G a u s s ia n G a u s s ia n=
Figure 6—8: Low Pass Filtering
XI. Restoration and Enhancement
( , )f x y → ( , ) '( , ) ( , )h x y h x y f x y→ →
The cascade of the two systems is the identity system.
1'( , ) min ,
( , )H u v A
H u v
=
,
Where, A is the maximum gain. Or more elegantly, we can use something like
59
2 2
( , )'( , )
( , )
H u vH u v
H u v B=
+,
where 1
(2 )Bis the maximum gain, if ( , )H u v is real.
XII. Correlation
Figure 6—9
Cross correlation of ( , )a x y and ( , )b x y is defined as
( , ) ( , )* ( , )
( , ) ( , )
ab x y a x y b x y
a x y b d d
φ
ε η ε η ε η∞ ∞
−∞ −∞
=
= − − ⋅∫ ∫
If ( , )a x y = ( , )b x y , the result is called autocorrelation
( , ) .( ( , ) ( , ))ab ab abx y Symmetric x y x yφ φ φ→ − − =
(0,0) ( , )aa aa x yφ φ≥ for all ( , )x y
2(0,0) ( , ) ( , ) ( , )aa a a d d a d dφ ε η ε η ε η ε η ε η∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
= ⋅ =∫ ∫ ∫ ∫
60
XIII. 6.13 Optimal Filtering
Figure 6—10
E = 2
( , ) ( , )o x y d x y dxdy
∞ ∞
−∞ −∞
−∫ ∫
( , ) ( , ) ( , )i x y b x y n x y= +
( , ) ( , ) ( , )o x y i x y h x y= ⊗
So, 2 2( , ) 2 ( , ) ( , ) ( , )E o x y o x y d x y d x y dxdy
∞ ∞
−∞ −∞
= − ⋅ +∫ ∫
22 ( , ) ( , ) ( , )o x y i x y h x y= ⊗
( , ) ( , ) ( , ) ( , )i x y h d d i x y h d dε η ε η ε η α β α β α β∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞
= − − ⋅ × − − ⋅∫ ∫ ∫ ∫
Therefore,
2 ( , ) ( , ) ( , ) ( , )o dxdy h h d d d d i x y i x y dxdyε η α β ε η α β ε η α β∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞
−∞ −∞ −∞ −∞ −∞ −∞ −∞ −∞
= ⋅ − − ⋅ − −∫ ∫ ∫ ∫ ∫ ∫ ∫ ∫
x A x A
y B y B
α α
β β
− = ⇒ = +
− = ⇒ = +
61
⇒ ( ( ), ( ) ( , )i A B i A B dAdBε α η β∞ ∞
−∞ −∞
= − − − − ⋅∫ ∫
( , )iiφ ε α η β= − − (Autocorrelation)
62
7. Image Processing – Discrete Image
I. Sampling Theorem
To convert a continuous distribution of brightness to a digital image, it necessary to
extract the brightness at each point arranged at a period. Thus is called sampling.
Assume an image as a one-dimensional function for simplicity.
Let the brightness at position x i.e. pixel value of pixel at x, be fix.
Figure 7—1
Figure 7—2
63
Suppose a function that is composed by infinite numbers of Dirac’s delta functions
assigned at an interval T. This is called comb function defined as follows.
A simple digital image from ( )f x , denoted ( )Tf x , is expressed as ( )f x multiplied by
the comb function ( )Tcomb x , i.e.
( ) ( ) ( )T Tf x f x comb x= ⋅
( ) ( ) ( ) ( ) ( ) ( )T TFT f x v FT f x v FT comb x v= ⊗
and, 11( ) ( ) ( )T
TFT comb x v comb V
T=
1
( ) ( ) ( ) ( )TFT f x v FT f x vT
= ⊗ 1( )
Tcomb V
That’s a convolution of comb function?
We start from “convolution with delta function”
( ) ( ) ( ) ( ) ( ) ( ) ( )f t t f y t y dy f t t t dy f tδ δ δ∞ ∞
−∞ −∞∗ = ⋅ − = ⋅ − =∫ ∫
i.e., convolution of a function and delta function is equal to original function itself.
Since a comb function is a sequence of the delta function arranged at constant period,
convolution of a function and comb function is an arrangement of duplication of the
original function at a constant period.
64
The F.T. of ( )Tf x , which is the sampled version of ( )f x at period T is an infinite
sequence of duplicating of ( )TFT f x which is the F.T. of the original function ( )f x
at period 1
T.
Figure 7—3
If the period of comb function in the frequency domain is sufficiently large, adjacent
( )FT f x ’s do not overlap. In this case, the F.T. of the original function ( )FT f x ,
can be separated and extracted i.e. no information of the brightness distribution of the
original image is lost by sampling. However, if the internal of the comb function is the
frequency domain is small, adjacent ( )FT f x (v)’s overlap. In this case, the original
( )FT f x can’t be separated and a faulty function will be extracted. This effect is
called aliasing.
65
( ) ( ) ( ) ( ) ( ) ( )T TFT f x v FT f x v FT comb x v⇒ = ⊗
Figure 7—4
Since the support of ( )FT f x is in the range between ~c cV V− , the period has to be at
least 2 cV to avoid overlapping of ( )FT f x ’s since, T is the sampling period. 1
T
denote the number of sampling per unit length, i.e. sampling rate consequently, the
original brightness distribution can be reconstructed by a sampled digital image of the
sampling rate is more than twice the maximum frequency contained in the original
distribution. This theorem is called sampling theorem.
66
Discrete Fourier Transform (DFT)
1 1 2 ( )2 ( )
0 00 0
1( , ) ( , ) ( , )
m nM N i u vi ux vy M N
m n
F u v f x y e dxdy f m n eMN
ππ
∞ ∞ − − − +− +
= =
= = ⋅∑∑∫∫
(u=0~M-1, v=0~N-1)
Inv. D.F.T.1 1 2 ( )
0 0
( , )m nM N i u vM N
m n
F u v eπ− − + +
= =
= ⋅∑∑ (m=0~M-1, n=0~N-1)
• f(m,n) ≜ Input Signal Amplitude (real or complex) at sample (m, n)
T ≜ Sampling interval
sf ≜ Sampling rate 1
( )T
= , samples/second
M,N = Numbers of Spatial samples = Number of frequency samples (integer)
• Magnitude 1
2 2 2( , ) ( , ) ( , )F u v R u v I u v = +
Phase Angle 1 ( , )( , ) tan
( , )
I u vu v
R u vφ −=
Power Spectrum 2 2 2( , ) ( , ) ( , ) ( , )P u v F u v R u v I u v= = +
• 1 1
0 0
1(0,0) ( , )
M N
m n
F f m nMN
− −
= =
= ∑∑ ⇒ Average of ( , )f m n or D.C. component.
If ( , )f x y is real, its F.T. is conjugate complex
( , ) ( , )
( , ) ( , )
F u v F u v
F u v F u v
∗
∗
= − −
= − − ⇒ Spectrum of D.F.T. is symmetric.
67
• Relationship between samples in the spatial and frequency domains.
1u
M x=
⋅ ∆
1v
N y=
⋅∆
• Translation
0 0
0 0
2 ( )
( , ) ( , )u v
i x yM Nf x y e F u u v v
π− ⋅ ⋅ + ⋅
⋅ ⇔ − −
0 0
0 0
2 ( )
( , ) ( , )ux vy
iM Nf x x y y F u v e
π− ⋅ +
− − ⇔
Case : when 0 0,2 2
M Nu v= =
2 ( )
( )2 2 cos ( ) ( 1)x y
ii x y x y
e e x yπ π π
⋅ ++ += = + = −
( , )( 1) ( , )2 2
x y M Nf x y F u v
+⇒ − ⇔ − −
And similarly,
( , ) ( , )( 1)2 2
u vM Nf x y F u v
+− − ⇒ −
Rotation cos
sin
x r
y r
θ
θ
= ⋅
= ⋅
cos
sin
u w
v w
φ
φ
= ⋅
= ⋅
Then, ( , )f x y and ( , )F u v become ( , )f r θ and ( , )F w φ
68
0 0( , ) ( , )f r F wθ θ φ θ+ ⇔ +
⇒ Rotating ( , )f x y by an angle 0θ rotates ( , )F u v by the same angle, and vice versa.
Periodicity ( , ) ( , ) ( , ) ( , )F u v F u M v F u v N F u M v N= + = + = + +
The inverse transform is also periodic.
( , ) ( , ) ( , ) ( , )f x y f x M y f x y N f x M y N= + = + = + +
Separability 1 1 22
0 0
1 1( , )
yxM M j vj uNM
x y
F u v e eM N
ππ− − −−
= =
= ⋅∑ ∑
1 2
0
1( , )
xM j uM
x
F u v eM
π− −
=
= ∑ , (where 1 2
0
1( , )
yM j vN
y
F u v eN
π− −
=
= ∑ )
( , ) ( , ) ( , )f x y F x v F u v→ →
• Convolution
1 1
0 0
1( , ) ( , ) ( , ) ( , )
M N
m n
f x y h x y f m n h x m y nMN
− −
= =
⊗ = ⋅ − −∑∑
f h F H
f h F H
⊗ ⇒ ⋅
⋅ ⇒ ⊗ Pr ( . .)ove H W→
1-D
Row transform
1-D
column transform
69
8. Edge & Edge detection
I. Edges in Images
• An Edge is a boundary between two figures with relatively distinct grey-level
properties.
• Edges are curves in the image where rapid changes occur in brightness or in the
spatial derivatives of brightness.
• Change of brightness can be occurred at
• Change of surface orientation
• Different surfaces
• One object occludes another
• A boundary between light and shadow falling on a single surface
II. Differential Operators
Figure 8—1
70
Figure 8—2
sin cos 0x yθ θ ρ− + =
( ) ( )1 2 1E , B B -B u( sin cos )x y x yθ θ ρ= + − +
If t > 0 B1
t < 0 B2
t = 0 ½(B1 + B2)
( ) ( )
( ) 1
t
u t x dx
x dx
δ
δ
−∞
∞
−∞
=
=
∫
∫
Figure 8—3
)cossin()(cos
)cossin()(sin
21
21
ρθθδθ
ρθθδθ
+−−−=∂
∂
+−−=∂
∂
yxBBy
E
yxBBx
E
71
squared gradient
2
21
22
))cossin()(( ρθθδ +−−=
∂
∂+
∂
∂yxBB
y
E
x
E
Laplacian of E(x,y)
2 22
2 2
22
1 22
22
1 22
2 '
1 2
sin ( ) '( sin cos )
cos ( ) '( sin cos )
( ) ( sin cos )
xx
yy
E EE
x y
EE B B x y
x
EE B B a y
y
E B B x y
θ δ θ θ ρ
θ δ θ θ ρ
δ θ θ ρ
∂ ∂∇ = +
∂ ∂
∂= = − − +
∂
∂= = − − +
∂
∇ = − − +
III. Discrete Approximations
• Discrete form of Laplacian
Ex = E( i+1 , j ) – E( i , j )
Ey = E( i , j+1 ) – E( i , j )
Exx = E( i – 1 , j ) – 2E( i , j ) + E( i + 1 , j )
Exx + Eyy = (Ei-1 j + Eij-1 + Ei-1 j + E i j+1 ) – 4Eij
72
IV. 8.4 Local operators and noise
bb
nn
H
φφ
ρ+
⋅−=1
12' -------------
ρ2 = u
2 + v
2
suppose that 2
2
2
& NS
nnbb == φρ
φ
Then 222
22'
NS
SH
ρ
ρ
+−= -
at low frequency -ρ2 : Laplacian operation
at high frequency -2
2
N
S << ∞ small gain Noise can be reduced !
Derivation of
Figure 8—4
''
' '' 2
1 1 ( )
1
( )
i bb nb
ii bb bn nb nn
bb
bb nn
nn
bb
H
SNR
H H
αϕ ϕ ϕ
ϕ ϕ ϕ ϕ ϕ
ϕ
ϕ ϕ
ϕϕ
+= =
+ + +
=+
= =+
= ⋅ ℑ ∇
73
For example, if the optimal filter is a gaussian
Figure 8—5
2
22
2
22
1),( σ
πσ
yx
eyxh
+−
=
)(2
1 222
)0,(vu
euH+−
=σ
74
9. Lightness & color
Figure 9—1
b´(x,y) = e´(x,y) - r´(x,y)
• Note that r´(x,y) is constant within a patch and sharp discontinuities at edges
between patches and e´(x,y) varies smoothly
Figure 9—2
Task : separate r´(x,y) & e´(x,y)
• Take the logarithm of image brightness b´(x,y)
log b´(x,y) = log r´(x,y) +log e´(x,y)
b(x,y) = r(x,y) +e(x,y)
∇2b(x,y) =∇2
r(x,y) +∇2e(x,y)
Figure 9—3
75
• cut slow slope by a threshold
Figure 9—4
T(∇2b(x,y) = ∇2
r(x,y)
|∇2b(x,y)| > T then t(x,y) = ∇2
b(x,y) else t(x,y) = 0
• recover r(x,y)
Figure 9—5
∫ ∫∇=
=∇
dxdyyxryxl
yxtyxl
),(),((
),(),(
2
2
Fourier Transform :
)(1
)(
)()(
)],([)],([
2
2
2
eTL
TL
yxtyxl
ρρ
ρρρ
−=⇒
=−⇒
ℑ=∇ℑ
2
1)(..
)()(
ρ
ρρ
−=
⋅=
pGwhere
TG
ρπ
+= )log(2
1)( rrg
IFT : I(x,y) = g(r) * t(x,y)
76
Figure 9—6
I(x,y) = r(x,y) = reflectance
e(x,y) = b(x,y) – I(x,y)
Figure 9—7
77
...
),(),(),(
),(),(),(
555
444
333
222
111
erb
erb
erb
yxeyxryxb
yxeyxryxb
⋅=
⋅=
⋅=
=
=
)(74
37: 21
22
11
2
11 ee
er
er
b
bR ≅== - equation 9.1
52
26
2
1 =r
r - equation 9.2
3
2
3
2
5
2
5
22
95
38
112
28:
r
r
b
b
r
r
b
bR
==
==
Least square error 11 variables 16 equations
0 ≤ reflectance ≤ 1
16
equations
78
10. Reflectance Map : photometric stereo
(using one camera)
I. Radiometry
irradiance A
pE
δ
δ= geometric stereo (using one camera)
radiance WA
pL
δδ
δ 2
=
solid angle = Area of a spherical surface patch / (radius of the sphere)2
Figure 10—1
2
2
14
2Solid angle 2
R
R
ππ
⋅= =
Figure 10—2
79
Figure 10—3
221
)1,,(ˆ
qp
qpn
++
−−=
II. Image Formation
Figure 10—4
80
solid angle = 2)cos/(
cos
α
θδ
z
O
solid angle = 2)cos/(
cos
α
α
S
SI
2
cos
cos
=
S
z
I
O
θ
α
δ
δ --------- equation 10.2.1
The solid angle subtended by the lens
ααπ
α
αα
π3
2
2
2
cos4)cos/(
cos2
=
=Ωzz
Thus the power of the light originating on the patch and passing through the lens is
θδδ cosΩ⋅= OLP
where L is the radiance of the surface in the direction toward the Lens
θααπ
δ
δ
δ
δ
θααπ
δδ
coscos4
coscos4
3
2
3
2
==
=
zI
OL
I
PE
zOLp
where E is the irradiance of the image at the patch under consideration
substituting for IO δδ , we finally obtain
81
ααπ
ααπ
4
2
4
2
cos4
cos4
⋅=
∝∴
=
sLE
LE
sLE
III. Bidirectional reflectance distribution function (BRDF)
Figure 10—5
Figure 10—6
82
),( iiE φθδ ; irradiance
The amount of light falling on the surface from the direction ),( ii φθ
),( eeL φθδ : radiance
The brightness of the surface as seen from the direction
),( ee φθ
BRDF :
1
cossin),;,(
),(
),(),;,(
2
0
=
=
∫ ∫−
π
π
π
φθθθφθφθ
φθδ
φθδφθφθ
eeeeeeii
ii
ee
eeii
dd
E
Lf
IV. Extended Light sources
Figure 10—7
solid angle wδ 2)tan(
.
cedis
areaThe = The area = iiδφθδθsin
83
(proof : 2
0
sin 2i i i
sw
ππ
π
θ δθ δϕ π−
= =∫ ∫ ∫ ∫
),( iiE φθ : Radiance per unit solid angle coming from the direction ),( ii φθ
The radiance from the patch under consideration
iiiiiii EE δφδθθφθδωφθ sin),(),( =
Total irradiance of the surface is
2
0
0
( , ) sin cosi i i i i i
E E d d
ππ
π
θ ϕ θ θ θ ϕ−
= ∫ ∫
where iθcos accounts for the foreshortening of the surface as seen from the direction
),( ii φθ
The radiance of the surface,
∫ ∫= ),(),,(),( iieeiiee EjfL φθφθφθφθ
iiii dd φθθθ cossin
V. Surface Reflectance properties
• Ideal Lambertian surface
One that appears equally bright from all viewing directions and reflects all
incident light, absorbing none
84
BRDF ),,( eeii jf φθφθ constant
2
0
2
0
( , , , ) sin cos 1
2 sin cos 1
1
1
i i e e e e e e
e e e
f j d d
f d
f
f
ππ
π
π
θ φ θ ϕ θ θ θ ϕ
π θ θ θ
π
π
−
=
=
=
∴ =
∫ ∫
∫
sine BRDF is constant for a Lambertian surface
)1
(1
cossin),(
ππ
φθθθφθ
=⋅=⋅=
⋅= ∫ ∫
fEEf
ddEfL
oo
iiiiii
∵
where the irradiance is Eo
• Ideal specular Reflector
Figure 10—8
85
2
0
( , ; , ) sin cos 1
sin cos 1
1
sin cos
i i e e e e e e
i i
i i
f d d
k
k
ππ
π
θ ϕ θ ϕ θ θ θ ϕ
θ θ
θ θ
−
=
⇒ =
⇒ =
∫ ∫
In this case, ii
ieieeeii jf
θθ
πφθδθθδφθφθ
cossin
)()(),,( −−−=
Determine the radiance of a specular reflecting surface under an extended source where
the irradiance is Eo
2
0
( ) ( )( , )
sin cos
( , ) sin cos
( , )
( , )
e i e ie e
i i
i i i i i i
e e
i i
L
E d d
E
E
ππ
π
δ θ θ δ ϕ ϕ πθ ϕ
θ θ
θ ϕ θ θ θ ϕ
θ ϕ π
θ ϕ
−
− − −=
⋅ ⋅
= −
=
∫ ∫
VI. Surface Brightness
Figure 10—9
86
iEL θπ
cos1
=
•Light source : a “sky” of uniform radiance E
2
0
2
0
2
0
1sin cos
sin cos
cos 2[1 1]
2 2
i i i i
i i i i
i
L E d d
Ed d
EE E
ππ
π
ππ
π
π
θ θ θ ϕπ
θ θ θ ϕπ
θ
−
−
= ⋅
=
− = = + =
∫ ∫
∫ ∫
The radiance of the patch ≡The radiance of the source
VII. Surface orientation
Figure 10—10
87
T
yyy
T
xxx
yx
qr
pr
qprrn
),,0(
),0,(
)1,,(
δδ
δδ
=
=
−−=×=
The unit surface normal n
221
)1,,(ˆ
qp
qp
n
nn
T
++
−−==
Figure 10—11
Tv )1,0,0(ˆ ≡
assuming α is very small
2222
22
1
1)1,0,0(
1
)1,,(ˆˆcos
1
)1,,(ˆ
qpqp
qpvn
qp
qpn
T
++=⋅
++
−−==
++
−−=
θ
88
VIII. The reflectance Map
Figure 10—12
),(ˆ
),(ˆ
22
11
qpRs
qpRs
→
→
iEL θπ
cos1
= for 0≥iθ
22222222 11
1
1
)1,,(
1
)1,,(ˆˆcos
sx
ss
ss
T
ss
i
qpqp
qqpp
qp
qp
qp
qpsn
++++
++=
++
−−⋅
++
−−=⋅=θ
= R(p,q) ; Reflectance Map
cqpqp
qqppqpR
ss
ss =++++
++=
2222 11
1),( - equation 10.9.1
Figure 10—13
89
222
1
2
1
222
222
1
2
1
11
1
11
1),(
11
1),(
ss
ss
ss
ss
qpqp
qqppqpR
qpqp
qqppqpR
++++
++=
++++
++=
IX. Shading in image
Consider a smoothly curved object the image of such an object will have spatial
variations in brightness due to the fact that surface patches with different orientations
appear with different brightness the variation of brightness is called shading
),(cos
cos4
4
2
qpRLE
f
dLE
i ≡
⋅=
θαα
απ
after normalizing E(x,y) = R(p,q)
ρ : reflectance factor where 10 ≤≤ ρ
nsyxEwhere
nspyxE
ˆˆ),(,1
)ˆˆ(),(
⋅==
⋅⋅=
ρ
Ex1) Consider a sphere with a Lambertian surface illuminated by a point source at
essentially the same place as the viewer in the case of cylinder
90
Figure 10—14
In this case,
221
)1,,(ˆ
)1,0,0(ˆ
)1,0,0(ˆ
qp
qpn
v
s
++
−−=
=
=
From equation 10.9.1 - snqpR ˆˆ),( ⋅=
22
2222
1
1
11
1
qp
qPqp
qqpp
ss
ss
++=
++++
++=
0
2
0
0
2
1
222
)(
)2())((2
1
zz
zzy
d
dq
zz
xxyxr
d
dp
y
z
x
z
−
−−==
−−=−+−==
−
Finally, E(x,y) = R(p,q)
91
2
22
2
0
22
22
1
)(1
1
1
1
r
yx
zz
yx
qp
+−=
−
++
=
++=
( x , y ) = ( 0 , 0 )
E( x, y) maximum = 0
x2+y
2 =r
2
E(x,y) minimum = 0
X. Photometric stereo
Figure 10—15
92
2
2
2
2
22
2222
2
1
2
1
22
1111
11
1),(
11
1),(
ss
ss
ss
ss
qPqp
qqppqpRE
qPqp
qqppqpRE
++++
++==
++++
++==
Ex2) R(p, q) : linear and independent
2
222
1
111
1),(
1),(
r
qqppqpR
r
qqppqpR
++=
++=
then 2121
12
2
221
2
1 )1()1(
pqqp
qrEqrEp
−
−−−=
2121
21
2
111
2
2 )1()1(
pqqp
prEprEq
−
−−−=
Ex3) R(p,q) : nonlinear
Figure 10—16
93
XI. Albedo : reflectance factor :
)ˆˆ( nsE ii ⋅= ρ
where 221
)1,,(ˆ
ii
T
ii
i
qp
qps
++
−−= for i=1, 2, 3, …………
221
)1,,(ˆ
qp
qpn
T
++
−−=
Figure 10—17
Unknown variables p, q, ρ
At least 3 equations
94
11. Reflectance Map : shape from shading
I. Recovering Shape from Shading
Figure 11—1
i. Picture
One more information => strong constraint smoothness neighboring patches of the
surface => similar orientation assumption of smoothness : strong constraint
ii. Linear reflectance map
Figure 11—2
95
220
2200 sincostan
ba
b
ba
a
a
b
+=
+== θθθ
Figure 11—3
:sincos)( θθθ qpm += directional derivative
)),((1
sincos)(
1
2222
000
yxEfbaba
bqap
qm
−
+=
+
+=
+=⇒ θθθ
S can be recovered from E(x,y) using above curve
Figure 11—4
96
δξδ mz = (small change a long the characteristic curve)
)),((1 1
22yxEf
ba
zm −
+==
δξ
δ
θξξ
θξξ
sin)(
cos)(
0
0
+=
+=
yy
xxwhere
0
0
1
02 2
0
( ) ( )
1( ( , ))
z z m d
z f E x y da b
ξ
ξ
ξ θ ξ
ξ−
= +
= ++
∫
∫
iii. Rotationally symmetric reflectance maps
Figure 11—5
97
22
22
sin
cos
tan
qp
q
qp
p
p
q
s
s
s
+=
+=
=
θ
θ
θ
The slope sss qpm θθθ sincos)( +=
22 qp +=
)),((1 yxEf −=
The change in Z
δξδξθδ
δξδξθδ
δξδξδ
22
22
122
sin
cos
)),((
qp
qy
qp
px
yxEfqpmz
s
s
+==
+==
⋅+== −
To simplify δz , we could take a step of length δξ22
1
qp + rather than δξ
δξ
δξ
δ
)),((
)(
1
22
2222
yxEf
qp
qpqpz
−=
+=
+×+=
δx = pδξ - equation 11.1.1
δy = qδξ
98
We need to determine p&q at the new point in order to continue the solution
yxtrwyxEateDifferenti
qpfyxE
&,,),(
)(),( 22 +=
22
' )(
qpSwhere
x
SSf
x
S
S
E
x
EEx
+=
∂
∂=
∂
∂⋅
∂
∂=
∂
∂=
y
zq
x
zp
x
x
pp
x
s
∂
∂=
∂
∂=
∂
∂+
∂
∂=
∂
∂,,22
yx
zq
x
zp
∂∂
∂+
∂
∂=
2
2
2
22
2 2
2
2( ) '( )
2( ) '( )
x
y
E pr qs f s
E ps qt f s
p p z zp x x y
q y x x yδ δ δ δ
= +
= +
∂ ∂ ∂ ∂= + = +
∂ ∂ ∂ ∂ ∂
- equation 11.1.2
ysxr δδ +=
ytxsq δδδ +=
in our case δξδδξδ qypx =&
δξδ
δξδ
⋅+=
⋅+=
)(
)(
qtpsq
qsprp
From (2)
δξδ
δξδ
'2
'2
f
Eq
f
Ep
y
x
=
=
As δξ → 0 , we obtain the differential equations
99
==
=+=== −
'2,
'2
)),((,, 122
f
Eq
f
Ep
yxEfqpzqypx
yx ɺɺ
ɺɺɺ
(x, y, z, p & q) solvable
5 Differential Eqs
one more time by differentiating
)),((
'2,
'2
..,,
1yxEfz
f
Eqy
f
Epx
trwqypx
yx
−=
====
==
ɺ
ɺɺɺɺɺɺ
ɺɺ ξ
iv. General Case
R(p,q) = f(ap + bq)
R(p,q) = f(P2 + q2)
R(p,q) = arbitarary
ysxr
yyx
zx
x
zp
yqxpz
δδ
δδδ
δδδ
+=
∂∂
∂+
∂
∂=
+=2
2
2
- equation 11.1.3
ytxs
yy
zx
yx
zq
δδ
δδδ
+=
∂
∂+
∂∂
∂=
2
22
100
⋅=
=
y
xH
y
x
ts
sr
q
p
δ
δ
δ
δ
δ
δ
where H is the Hessian Matrix
2
2
2
2
y
z
x
ztr
∂
∂+
∂
∂=+ : Laplacian
Differentiating E(x,y) w.r.t x & y
x
q
q
E
x
p
p
E
x
EEx
∂
∂⋅
∂
∂+
∂
∂⋅
∂
∂=
∂
∂=
SRrR
Syx
zR
x
zR
qp
qp
⋅+⋅=
∂∂
∂⋅+
∂
∂⋅=
2
2
2
tRSRE qpy ⋅+⋅=
⋅=
q
p
y
x
R
RH
E
E - equation 11.1.4
from equation 11.1.3 & equation 11.1.4
Let δξδ
δ⋅
=
q
p
R
R
y
x
q
p
Ry
Rx
=
=
ɺ
ɺ
Then δξδ
δ
δ
δ
=
⋅=
⋅=
y
x
q
p
E
E
R
RH
y
xH
q
p
101
==
+=+=⋅+⋅=
+==
=
yx
qp
qpq
p
EqEp
qRpRyqxpzy
qx
pz
qRpRzRy
Rx
ɺɺ
ɺɺɺ
ɺɺ
ɺ
,
,
δξ
δ
δξ
δ
δξ
δ
5unknown variables : x,y,z, p & q
Figure 11—6
II. 11.3 Singular points
Figure 11—7
),(),( 00 qpRqpR < for all ),(),( 00 qpqp ≠
Figure 11—8
102
0,0
0,0
====
==
qp
qp
RyRx
RR
ɺɺ
)(2
1),( 22
qpqpR += - equation 11.3.1
(p,q) = (0,0) – singular point
)2(2
1 22
0 cybxyaxzz +++=
Thus byaxdx
dzp +==
& cybxdy
dzq +==
substituting p & q in equation 11.3.1
])()[(2
1)(
2
1),( 2222
cybxbyacqpqpR +++=+=
222222 )(2
1)()(
2
1ycbbxycaxba +++++=
The brightness gradient is
ycbbxcaE
bycaxbaE
y
x
)()(
)()(
22
22
+++=
+++=
Differentiating again,
+=
+=
+=
22
22
)(
cbE
bcaE
baE
yy
xy
xx
3 equations, 3 unknowns
103
III. Stereo graphic projection
Figure 11—9
=+
+=
+=
+
222
22
22
)(
2
Rba
R
qp
a
b
R
qf
aR
b
solve the above equations
2222 11
2,
11
2
qp
qg
qp
pf
+++=
+++=
conversely
2222 4
4,
4
4
gf
gq
gf
fp
−−=
−−=
104
IV. 11.7 Relaxation methods
characteristic strip expansion
E(x , y) = R(p , q)
= Rs(f,g)
Minimize
∫ ∫ +++= dxdygygxfyfes )()( 2222
A low, Minimize
∫ ∫ −= dxdygfRyxEe si
2)),(),((
overall, minimize is ee λ+
if brightness measurement are very accurate,
λis large
if brightness measurement are very noisy,
λis small
105
minimize ∫ ∫ dxdyggffgfF yxyxI ),,,,,(
= It’s a problem in the calculus of variations.(see appendix)
The corresponding Euler equations are
0
0
=∂
∂−
∂
∂−
=∂
∂−
∂
∂−
yx
yx
ggg
fff
Fy
Fx
F
Fy
Fx
F
where Ff is the partial derivative of F with respect to F
22222)),(),(()()( gfRyxEggffF syxyx ⋅++++= λ
The Euler equations for this problem yield
yyf
xxxf
s
sf
fFy
ffx
Fx
f
RgfRyxEF
y
x
2
2)2(
)),(),((2
=∂
∂
=⋅∂
∂=
∂
∂
∂
∂−−= λ
fffyyxx FFy
Fx
ffyx
=∂
∂+
∂
∂=+ )(2
f
RgfRyxE s
s∂
∂−−= )),(),((λ
g
RgfRyxEg s
s∂
∂−−=∇ )),(),((2 λ (2 equations, 2 unknowns(f,g))
Application to photometric stereo consider n images,
106
∫ ∫ ∑ ∫ ∫=
−+++−=n
i
iiIiyxyI dxdygfRyxEdxdyggffxe1
22222 )),(),(())()(( λ
where Ei(x,y) is the brightness measured in the i th image and Ri is the corresponding
reflectance map
The Euler equations for this problem yield
∑
∑
=
=
∂
∂−−=∇
∂
∂−−=∇
n
i
i
iii
i
ii
n
i
i
g
RgfRyxEg
f
RgfRyxEf
1
2
1
2
)),(),((
)),(),((
λ
λ
V. 11.8 Recovering Depth from a needle diagram
Figure 11—10
107
Figure 11—11
0 0
( , )
0 0
( , )
( , ) ( , ) ( )
x y
x y
z x y z x y pdx qdy= + +∫
Minimize the error
dxdypzpzI
px ))()(( 22 −+−∫ ∫
where p and q are the given estimates of the components of the gradient while yx zz &
are the partial derivatives of the best-fit surface
Minimize ∫ ∫ dxdyzzzF yxI ),,(
The Euler equations is
0=∂
∂−
∂
∂−
yx zzz Fy
Fx
F
108
so that from 22 )()( qzpzF yx −+−=
0
)(2)(2
=
−∂
∂⋅+−
∂
∂⋅=
∂
∂+
∂
∂=
qzy
pzx
Fy
Fx
F
yx
zzz yx
)(
0)(2)(2
2
yx
yxyyxx
yyyxxx
qpz
qpzz
qzpz
+=∇
+=+
=−+−
109
12. Motion field & Optical flow
Figure 12—1
• Motion field : a purely geometric concept
• Optical flow : Motion of brightness patterns
observed when a camera is moving relative to the objects being imaged
I. Motion Field
Figure 12—2
110
0
0
0
0
,&
ˆ'
ir r
i
t t
i
v v
r r
f r z
δ δ
δ δ= =
=⋅
- equation 12.1.1
Differentiate - equation 12.1.1
2
0
0000
2
0
000
0
)ˆ(
)ˆ()ˆ(
)ˆ(
)ˆ()ˆ(
'
1
zr
rzvvzr
zr
rzdt
dr
dt
drzr
dt
rd
f
i
⋅
⋅−⋅=
⋅
⋅−⋅=⋅
2
0
00
)ˆ(
ˆ)(
'
1
zr
zvrv
fi
⋅
××=⋅ - equation 12.1.2
II. Optical flow
Figure 12—3
111
Ex1)
Figure 12—4
E(x, y, t) ≡ E(x + uδt , y + vδt , t + δt) where u(x, y) & v(x, y) are the x & y components
of the optical flow vector
For a small time interval tδ , if brightness varies smoothly with x, y, &t,
),,(),,(),,( tyxEt
Et
y
Ey
x
ExtyxEtttvytuxE ≡++++=+++ ρ
α
αδ
α
αδ
α
αδδδδ
where ρ contains second and higher order terms in δx , δy , and δt
0=⋅∂
∂+⋅
∂
∂+⋅
∂
∂
t
t
t
E
t
y
y
E
t
x
x
E
δ
δ
δ
δ
δ
δ
0=++ tyx EvEuE (Optical flow constraint equation)
u, v : two variables
one equation :
⇒ one more constraint needed
III. Smoothness of the optical flow
Usually, the motion field varies smoothly minimize a measure of departure from
smoothness
112
∫ ∫ +++= dxdyvyvxuyuxes ))()(( 2222
Also, minimize the error in the optical flow constraint eq.
∫ ∫ ∫∫ ++=−= ++ ydxdEvEuEdxdytyxEtyxEe tyxatttatc
22 )(]),,(),,([ δ
Hence, minimize cs ee λ+ ,
Minimize cs ee λ+
Where ∫ ∫ +++= dxdyvyvxuyuxes ))()(( 2222
∫ ∫ ++= dxdyEvEuEe tyxc
2)(
λ: a parameter that weights the error in the image motion equation relative to the
departure
from smoothness
λ=> Large
=> small
Minimize ∫ ∫ dxdyvvuuvuF yxyx ),,,,,(
22222 )()()( tyxyxYX EvEuEvvuuF ++++++= λ
Euler equations are
0
0
=∂
∂−
∂
∂−
=∂
∂−
∂
∂−
yx
yx
vvv
uuu
Fy
Fx
F
Fy
Fx
F
113
yyyu
xxxu
xtyxu
uux
Fy
uux
Fx
EEvEuEF
y
x
2)2(
2)2(
)(2
=∂
∂=
∂
∂
=∂
∂=
∂
∂
++= λ
uuuEEvEuE yyxxxtyx
2)( ∇=+=++λ
Similarly,
0
)( 2
=++
∇=++
tyx
ytyx
EvEuE
vEEvEuEλ
IV. Boundary conditions
*Natural boundary conditions (see appendix)
ds
dxF
ds
dyF
ds
dxF
ds
dyF
yx
yx
vv
uu
=
=
where S denotes arc length along the boundary curve
T
ds
dx
ds
dyn
−= ,ˆ is a unit vector
perpendicular to the boundary
Rewriting the above conditions.
0ˆ),(
0ˆ),(
=⋅
=⋅
nFF
nFF
yx
yx
vv
uu
114
in our case, 22222 )()()( tyxyxyx EvEuEvvuuF ++++++= λ
0ˆ),(
0ˆ),(
=⋅
=⋅
nvv
nuu
yx
yx
V. Discrete case
From cs eee λ+=
dxdyEvEuEdxdyvvuu tyxyxyx
22222 )())()(( ++++++= ∫ ∫∫ ∫ λ
)( ij
i j
ij cse λ+=∑∑
))()()()((4
1 2
,1,
2
,,1
2
,1,
2
,,1, jijijijijijijijiji vvvvuuuus −+−+−+−= ++++
2)( tijyijxij EvEuEc ++=
Differentiating w.r.t klkl vu & yields
,)(2)(2
,)(2)(2
ytklyklxklkl
kl
xtklyklxklkl
kl
EEvEuEvvv
e
EEvEuEuuu
e
+++−=∂
∂
+++−=∂
∂
λ
λ
where vu & are average of u & v
115
0
0
=∂
∂
=∂
∂
kl
kl
v
e
u
e
solve this
tyklxklxyklyx
txklyxklyklyx
EEvEuEEvEE
EEvEEuEuEE
λλλλ
λλλλ
−++−=++
−−++=++
)1())(1(
,)1())(1(
222
222
Iterative scheme is
.)(1
,)(1
22
1
22
1
y
yx
t
n
kly
n
klxn
kl
n
kl
x
yx
t
n
kly
n
klxn
kl
n
kl
EEE
EvEuEvv
EEE
EvEuEuu
++
++−=
++
++−=
+
+
λ
λ
Figure 12—5
116
13. Photogrammetry & Stereo
I. Disparity between the two images.
Figure 13—1
Disparity : Dxx rl =− ''
)3(
)2()(2
)1()(2
''
'
'
−==
−←−
=
−←+
=
z
y
f
y
f
y
lineredz
bx
f
x
linebluez
bx
f
x
rl
r
l
solve (1) (2) & (3)
fz
bxx rl =− )( '' : disparity
''''
''
''
''
,2/)(
,2/)(
rlrl
rl
rl
rl
xx
bfz
xx
yyby
xx
xxbx
−=
−
−=
−
−=
117
Dxxz
rl
11''
=−
∝
II. Photogrammetry
Figure 13—2
• absolute orientation
),,( knownrr rl
r
r
r
z
y
x
0rrRr lr +=
333231
232221
131211
rrr
rrr
rrr
component
rotaional
34
24
14
r
r
r
component
naltranslatio
xr
118
where R is a 3*3 orthonormal matrix
IRRT =⋅
rlll
rlll
rlll
zrzryrxr
yrzryrxr
xrzryrxr
=+++
=+++
=+++
34333231
24232221
14131211
3 equations => 12 unknowns
• Orthonormality
IRRT =⋅
1
1
1
2
33
2
32
2
31
2
23
2
22
2
21
2
13
2
12
2
11
=++
=++
=++
rrr
rrr
rrr
0
0
0
133312321131
332332223121
231322122111
=++
=++
=++
rrrrrr
rrrrrr
rrrrrr
Figure 13—3
119
1212 rrll rrrr −=−
3+7 = 10 equations
12 unknown variables
P&Q : 10+3 = 13 equations
Figure 13—4
0rrRr lr +⋅= - equation 13.3.1
=
=
34
24
14
0
333231
232221
131211
r
r
r
r
rrr
rrr
rrr
R
III. Relative orientation
rl rr , ; not known rl zz , unknowns
The projection points ),(&),( ''''
rrll yxyx known determine the ratios of x and y to z
using
120
r
rr
r
rr
l
ll
l
ll
z
y
f
y
z
x
f
x
z
y
f
y
z
x
f
x
==
==
''
''
&
&
- - equation 13.4.1
12 + 2*(n)
substitute B into A
l
r
rl
ll
l
l
rlll
z
f
f
zxrzr
f
zyr
f
zxr
z
f
xrzryrxr
)()'( '
141312
'
11
14131211
=+++
=+++
⇒
==+++
==+++
==+++
l
r
l
ll
r
l
r
r
l
ll
r
l
r
r
l
ll
z
zf
z
frfryrxr
s
y
z
zy
z
frfryrxr
s
x
z
zx
z
frfryrxr
3433
'
32
'
31
'
2423
'
22
'
21
'
1413
'
12
'
11
- equation 13.3.2
12 + 2n ; # of unknowns for n points
7 + 3n ; # of equations
7 from orthonormality & distance length
121
IRRT =⋅
100
010
001
jiifrr
r
ji
i
≠=
=
0
12
∴12 + 2n = 7 + 3n
n = 5
n ≥5
IV. 13.5 Using a known relative orientation
sf
z l = ; constant sxf
zxx l
l
ll ⋅== /' syy ll ⋅= ' sfz l ⋅=
where s is a scaling factor
from equation 13.3.2
wcsrsfryrxrz
vbsrsfryrxry
uasrsfryrxrx
llr
llr
llr
+=+++=
+=+++=
+=+++=
3433
'
32
'
31
2423
'
22
'
21
1413
'
12
'
11
)(
)(
)(
wcs
vbs
z
y
f
y
wcs
uas
z
x
f
x
r
rr
r
rr
+
+==
+
+==
'
'
122
V. Computing depth
Once IR , ),(),,(, ''''
0 rrll yxyxr known, The depth ),( rl zz can be computed
as follows
Figure 13—5
From equation 13.3.2, multiply by f
z l
rl
ll
r
r
l
ll
r
r
l
ll
zrzrf
yr
f
xr
zf
yrzr
f
yr
f
xr
zf
xrzr
f
yr
f
xr
=+++
=+++
=+++
3433
'
32
'
31
'
2423
'
22
'
21
'
1413
'
12
'
11
)(
)(
)(
From any two equations
Compute rl zz &
Then compute
r
rrT
rrrl
l
llT
llll
zf
y
f
xzyxr
zf
y
f
xzyxr
==
==
1,,),,(
1,,),,(
''
''
123
VI. 13.7 exterior orientation
Figure 13—6
0rrRr ac +⋅=
caaa
caaa
caaa
zrzryrxr
yrzryrxr
xrzryrxr
=+++
=+++
=+++
34333231
24232221
14131211
IRRT =⋅ (orthonormality)
c
c
c
c
z
y
f
y
z
x
f
x==
'&
'
34333231
24232221
34333231
14131211
'
'
rzryrxr
rzryrxr
z
y
f
y
rzryrxr
rzryrxr
z
x
f
x
aaa
aaa
c
c
aaa
aaa
c
c
+++
+++==
+++
+++==
- equation 13.7.1
# of equations = 2n +6
# of unknowns = 12
2n+6 = 12 => n=3 (at least we need three equations)
124
VII. Interior Orientation
• scaling error
• translation error
• skewing error
• shearing error
An affine transformation
232221
131211
)()('
)()('
az
ya
z
xay
az
ya
z
xax
c
c
c
c
c
c
c
c
++=
++=
from equation 13.7.1
34333231
24232221
34333231
14131211
'
'
szsysxs
szsysxs
f
y
szsysxs
szsysxs
f
x
aaa
aaa
aaa
aaa
+++
+++=
+++
+++=
ijsR : not orthonormal
IRRijij s
T
s ≠⋅ 12 unknowns
#of equations = 2n
∴ 2n=12 ; n=6
125
VIII. Finding conjugate points
i. Gray-Level Matching
• (reference figure 3-1)
z
bx
f
x
z
bx
f
x rl 2&2'' −
=+
=
at a matched point
),(),( ''''
rrrlll yxEyxE =
)(
)',2()',2(~
''yyy
yz
bx
fEyz
bx
fEE
rl
rl
=≅
−=
+
∵
Let z
bfyxd
z
x
f
x=≅ )','(&
'
)'),','(2
1'()'),','(
2
1'( yyxdxEyyxdxE rl −=+
criterion : minimize is eee λ+=
where ∫ ∫ ∇⋅∇= '')( 2dydxdes
∫ ∫ −= '')( 2dydxEEe rli
(solution) using Euler equation
0''
'' =∂
∂−
∂
∂−
yx dd Fy
Fx
Fd
where
126
( )
]''
[2
1()]()[2
]'
())
2
1()
2
1(
'
()()][()[2
)]'),','(2
1'()'),','(
2
1'([)( 22
x
E
x
EEE
x
E
x
EEEEFd
yyxdxEyyxdxEdF
rl
rl
rl
rl
rl
∂
∂+
∂
∂−=
∂
∂−−
∂
∂−−=
−−++∇⋅∇=
λ
λ
λ
]'
()
'
()[
2
1()]()[)(
)(2''
22
22
''
x
E
x
EEEd
dFy
Fx
rl
rl
dydx
∂
∂+
∂
∂−=∇∇
∇∇=∂
∂+
∂
∂
λ∵
127
14. Pattern classification
Reference – “ Pattern Classification & Scene Analysis”
Dudda & Hart (Wiley Interscience pub. C)
1. Bayes Decision Theory
Fundamental statistical approach to the problem of pattern classification
• Maximum likelihood estimation : parameters fixed but unknown
• Baysian estimation : parameters, random variables having some known a prior
distribution
(A) Maximum likelihood estimation
Figure 14—1
Given ; 1. Model of system
2. pata : outcome of some type of probabilistic experiment.
Goal : Estimate unknown model parameters from data
Let nii .,.........2,1, =λ be n I.I.D. observations on a random variable X drawn from
):( θxf x
128
The joint lpdf of nXXX ,........, 21 is 1xf
Where );..,.........( 1 θnxxL is called the likelihood function (LF)
The maximum likelihood estimation (MLE) θ is that value that maximizes the LF, that
is
);..,.........()ˆ;..,.........( 11 θθ nn xxLxxL ≥
Ex1) Assume ),(; 2σµNX where σis known. Compute the MLE of the mean μ
<solution> The LF for h realizations of X is on a
−−
= ∑
=
h
i
i
n
xXL0
2
22)(
2
1exp
2
1);( µ
σπσµ
since the log function is monotonic, the maximum of L(x ; μ) is also that of log L(x ; μ)
Hence ∑=
−−−=n
i
ixn
xL1
2
2
2 )(2
1)2log(
2);(log µ
σπσµ
Set 0);(log
=∂
∂
u
xL µ
This yields ∑=
=−∂
n
i
ix0
20)(
1µ ∑
=
=n
i
ixn 1
1µ
Ex2) Consider the normal p.d.f.,
129
∑=
−−=n
i
x xxf1
2
22
2 ))(2
1exp(
2
1),,( µ
σπσσµ
The log likelihood function is
∑=
−−−−=n
i
ixnn
jxxL1
2
2
2
21 )(2
1log2log
2),,.......(log µ
σσπσµ
Now set 0,0 =∂
∂=
∂
∂
σµ
LL
Obtain the simultaneous equations
∑
∑
=
=
=−+−
=−
n
i
i
n
i
i
xn
x
1
2
3
1
0)(1
0)(
µσσ
µ
Hence ∑=
=n
i
iXn 1
1µ
∑=
−=n
i
ixn 1
22 )ˆ(1
ˆ µσ
(B) Baysian Estimation
Suppose that an classifier to separate two kinds of lumber, ash and birch
Figure 14—2
130
A prior : • probability p(w1) : the next piece is ash
• probability p(w2) : the next piece is birch
Decision Rule : Decide w1 if p(w1) > p(w2)
Decide w2 otherwise
Bay’s Rule ;
)(
)()|()|(
xp
wpwxpXwp
jj
j
⋅=
where ∑=
⋅=r
j
jj wpwxpxp1
)()|()(
Bayes Decision Rule for minimizing the probability of error ;
Decide w1 if p(w1|x) > p(w2|x)
Decide w2 otherwise
Decide w1 if P(x| w1)p(w1) > p(x| w2)p(w2)
Decide w2 otherwise
If equally likely, p(w1)=p(w2), then
131
Decide w1 if p(x| w1) > p(x| w2)
Decide w2 if otherwise
Figure 14—3
Probability error : p(error)
( ) ( , )
( | ) ( )
p error p error x dx
p error x p x dx
∞
−∞
∞
−∞
=
=
∫
∫
where p(error|x)
12
21
)|(
)|(
wdecideweifxwp
wdecideweifxwp
Figure 14—4
132
(B-1) Generalized Form (for Bayes classifier)
Let .......... 1 sww=Ω be the finite set of S states of nature and )........ 1 aA αα= be the
finite set of possible actions.
Let )|( ji wαλ be the loss incurred for taking action iα when the state of nature is jw
Let the feature vector X be a d-component vector-valued r.v
Then )(
)()|()|(
xp
wpwxpxwp
jj
j =
Where ∑=
=s
j
jj wpwxpXp1
)()|()(
The expected loss associated with taking action iα is
)|()|()|(1
∑=
⋅=s
j
jjii XwpwXR αλα
Target: to find a Bayes decision rule against p(wj) that minimizes the overall risk
The overall risk is given by
∫ ⋅= xdxpxxRR )()|)((α
clearly, if )(xα is chosen so that )|)(( xxR α is as small as possible, for every X , then
the overall risk will be minimized Bayes decision rule for the general form; to minimize
the overall risk.
133
1.Compute the conditional risk
∑=
⋅=s
j
jjii XwpwXR1
)|()|()|( αλα
for i=1,………..a
2.Select the action iα for which )|( XR iα is minimum
3.Obtain the resulting minimum overall risk called the Bayes risk
B.2 Two category classification
Let )|( jiij wαλλ = the loss incurred for deciding wi when the true state of nature is wj.
)|()|()|(
)|()|()|(
2221212
2121111
XwpXwpXR
XwpXwpXR
λλα
λλα
+=
+=- equation 14.1
The decision rule :
Decide w1 if )|()|( 21 XRXR αα < - equation 14.2
Decide w2 otherwise
If we substitute 1 into 2
We have
)|()()|()(
)|()|()|()|(
1112122212
222121212111
XwpXwp
XwpXwpXwpXwp
λλλλ
λλλλ
−<−⇒
+<+
Hence likelyhood ratio
134
Figure 14—5
<Minimum – error – rate classification>
Symmetrical on zero-one loss function
The loss )|( ji wαλ = 0 i=j for i,j=1,……c
1 i≠j
all errors are equally costly
The conditional risk is
∑=
⋅=c
j
jjii XwpwXR1
)|()|()|( αλα
)|(1
)|(
Xwp
Xwp
i
c
ij
j
−=
=∑≠
To minimize the average prob of error, we should select the i that maximizes the a
posterior prob
)|( Xwp i In other words, for minimum error rate j
Decide iw if )|()|( XwpXwp ji > for all ,ij ≠
B-3 The multicategory classification
135
A pattern classifier
Figure 14—6
),(Xg i for i=1,………c ; discriminant function
The classification is said to assign a feature vector X to class jw
If )()( XgXg ji > for all ij ≠
)|()( XwpXg ii = ; a posterior prob
∑=
=c
j
jj
ji
wpwXp
wpwXp
1
)()|(
)()|( by Bayes rule
)(log)|(log
)()|(
ii
ii
wpwXp
wpwXp
+⇒
⇒
Decision Rule
1. Divide the feature space into C division regions, R1,………..Rc
2. If )()( XgXg ji > for all j≠i then X is in RI
136
In other words, assign X to wI
Ex) three category classification
Probability error calculation
P(error Xo) )()( 00 XPXerrorP ⋅=
)()()()( 330220 wPwxPwPwXP +=
Total error
P(error) ∫∞
∞−= dxxerrorP ),(