computer and robot vision ii

Digital Camera and Computer Vision LaboratoryDepartment of Computer Science and Information Engineering

National Taiwan University, Taipei, Taiwan, R.O.C.

Computer and Robot Vision II

Chapter 18Object Models And Matching

Presented by: 傅楸善 & 徐子凡0989306249

[email protected]指導教授 : 傅楸善博士

mailto:[email protected]

DC & CV Lab.DC & CV Lab.CSIE NTU

18.1 Introduction

object recognition: one of most important aspects of computer vision


Joke


18.2 Two-Dimensional Object Representation

2D shape analysis useful in machine vision application: medical image analysis aerial image analysis manufacturing


18.2 Two-Dimensional Object Representation

2D shape representation classes: 18.2.1 global features 18.2.2 local features 18.2.3 boundary description 18.2.4 skeleton 18.2.5 2D parts


18.2.1 Global Feature Representation

2D object: can be thought of as binary image

value 1: pixels of objectvalue 0: pixels outside object

2D shape features: area, perimeter, moments, circularity, elongation



Shape Recognition by Moments : binary image function : 2D shape digital th moment of :

area of S number of pixels of S

1,|, yxfyxS

kj,

Syx

kjjk yxM

,

:00 SM

f

S



moment invariants are functions of digital moments invariant under certain

shape transformations. translation, rotation, scaling, skew

center of gravity of S: yx,

)(

)(

00

01

00

10

SM

SMy

SM

SMx

Syx

kjjk yxM

,



central th moment of S:

central moments: translation invariant normalized central moments of S:

),( kj

Syx

kjjk yyxx

),(

)()(

12

,00

kjjk

jk

)(

)(

00

01

00

10

SM

SMy

SM

SMx



seven functions that are rotation invariant


Original

Half Size Mirrored Rotated 2° Rotated 45°



Fourier descriptors: another way for extracting features from 2D shapes defined to characterize boundary

The main idea is to represent the boundary as a function of one variable , expand in its Fourier series, and use the coefficients of the series as Fourier descriptors (FDs).

finite number of FDs: can be used to describe the shape

t t



Each coordinate pair can be treated as a complex number so that

Discrete Fourier transform

of is

( ) ( ) ( ),

for 0,1,2,..., 1

s k x k jy k

k K

( )s k21

0

1( ) ( ) ,

for 0,1,2,..., 1

j ukKK

k

a u s k eK

u K



The complex coefficients are called the Fourier descriptors of the boundary.

The inverse Fourier transform of these coefficients restores .

Suppose, only the first P coefficients are used.

( )a u

( )s k21

0

( ) ( ) ,

for 0,1,2,..., 1

j ukKK

u

s k a u e

k K

21

0

ˆ( ) ( )j ukP

K

u

s k a u e



Some basic properties of Fourier descriptors.

Notation:

Impulse function :xy x j y

( ) 0, if 0

( ) 0, if 0

u u

u u

( )u


Joke


18.2.2 Local Feature Representation

2D object characterized by: local features, attributes, relationships

most commonly used local features: Holes

found by connected component procedure followed by boundary tracing

detected by binary mathematical morphology, if hole shapes known

properties: areas, shapes Corner

detection: can be performed on binary or gray tone image property: angle at which lines meet


Joke


18.2.3 Boundary Representation

boundary representation: most common representation for 2D objects.

3 main ways to represent object boundary: sequence of points chain code sequence of line segments


18.2.3 Boundary Representation The Boundary as a Sequence of Points boundary points from border-following or edge-

tracking algorithms interest points: boundary points with special property

useful in matching



The Chain Code Representation chain encoding:

can be used at any level of quantization saves space required for row and column coordinates

boundary encoded: first quantized by placing over square grid

square grid side length: determines resolution of encoding

marked points: grid intersections closest to curve and used in encoding

＊ : marks starting point of curve


chain encoding of boundary curve



line segments: links: to be used to approximate the curve

encoding scheme: eight possible directions assigned integer between 0, 7

chain: chain encoding: in the form

i

n

in aCAoraaaA

121 ...



length of chain code with n chains: can be simply estimated as n

no: number of odd chain codes

ne: number of even chain codes

nc: number of corners L: unbiased estimate of perimeter length Freeman suggested: oe nnL 2


18.2.3 Boundary Representation The Boundary as a Sequence of Line Segments line segment sequence: after boundary segmented

into near-linear portion line segment sequence: used in shape

recognition or other matching tasks : coordinate location where pair of lines

meet : angle magnitude where pair of lines meet sequence of junction points to

represent line segment sequence

:, ii YX

i:,...,, 21 nOOOO

iii YX ,,



sequence of junction points representing test object T

an association

goal: given O, T, to find F satisfying i < j F(i) < F(j) or F(i) = missing or F(j) = missing

:,...,, 21 mTTTT

:missing,...,2,1,...2,1: nmF


Joke


18.2.4 Skeleton Representation

strokes: long, sometimes thin parts forming shapes



symmetric axis transform: set of maximal circular disks that fit inside object

symmetric axis: locus of centers of these maximal disks



The symmetric axis is one example of a skeleton description of 2D object.

symmetric axis is not always completely representative of the strokes of an object. rectangle: consists of five line segments not

single line symmetric axis: extremely sensitive to noise

make it difficult to use in matching.



local symmetry: midpoint P of line segment BA

α : angle between BA and outward normal Na at A

α : angle between BA and inward normal Nb at B



The loci of local symmetries that are maximal w.r.t. forming a smooth curve are called axes or spines.

cover of axis: portion of shape subtended by axis axis cover properly contained in another cover:

second axis subsumes first

The short diagonal axes are subsumed by the horizontal and vertical axes and can be either deleted or relegated to a lower place in a hierarchical description of the shape (Chap. 19).



Axes of smoothed local symmetries of several objects.


Joke


18.2.5 Two-Dimensional Part Representation

parts, attributes, interrelationships: form structural description of shape

nuclei: regions where primary convex subset overlap

nuclei



near-convexity: allows noisy distorted instances to have same decompositions

P1 , P2: two points on object boundary

LI relation: visibility relation if line completely interior to object boundary,

Apply the graph-theoretic clustering algorithm to determine clusters of visibility relation

1 2PP

ILPP 21,



decomposition of three similar shapes into near-convex pieces


Joke


18.3 Three-Dimensional Object Representations

18.3.1 Local Features Representation. 18.3.2 Wire Frame Representation. 18.3.3 Surface-Edge-Vertex Representation. 18.3.4 Stick, Plates, and Blobs. 18.3.5 Generalized Cylinder Representation. 18.3.6 Super-quadric Representation. 18.3.7 Octree Representation. 18.3.8 The Extended Gaussian Image. 18.3.9 View-Class Representation.


18.3.1 Local Features Representation

Local Features Representation range data:

obtained from laser range finder, light striping, stereo, etc. from depth, try to infer surfaces, edges, corners, holes, other

features 3D matching more difficult than 2D because of occlusion


Joke


18.3.2 Wire Frame Representation

wire frame model: 3D object model with only edges of object



two-color hyperboloid and its line drawing



Necker cube: lower-vertical face or upper-vertical face closer to viewer

Schroder staircase: viewed either from above or from below



general-viewpoint assumption: none of the following situations 1. two vertices of scene objects represented at same picture

point 2. two scene edges seen as single line in picture 3. vertex seen exactly in line with unrelated edge



general-viewpoint assumption: heart of line-drawing interpretation

viewpoint in perspective projection: center of projection

viewpoint in orthographic projection: direction of projection


subjective contours of Kanizsa: white occluding triangle in space




line labels for visible projections of surface-normal discontinuities:


Joke


18.3.3 Surface-Edge-Vertex Representation

VISIONS system: Visual Integration by Semantic Interpretation of Natural Scenes

PREMIO system: Prediction in Matching Images to Objects

PREMIO 3D object model: hierarchical, relational model with five levels world, object, face/edge/vertex, surface/boundary, arc/2D,

1D piece



world level: arrangement of different objects in world object level: arrangement of different faces, edges,

vertices forming objects face level: describes face in terms of surfaces and

boundaries surface level: specifies elemental pieces forming

surfaces



2D piece level: describes pieces and specifies arcs forming boundaries

1D piece level: describes elemental pieces forming arcs

SDS: spatial data structure A/V: attribute-value table


Joke


18.3.4 Sticks, Plates, and Blobs

sticks, plates, blobs model: rough models of 3D objects used in rough-matching near-convex

sticks: long, thin parts with only one significant dimension cannot bend very much two logical endpoints set of interior points center of mass



plates: flattish wide parts with two nearly flat surfaces two significant dimensions cannot fold very much set of edge points, set of surface points, center of mass

blobs: parts with three significant dimensions can be bumpy but cannot have concavities set of surface points and center of mass



attribute-value table: contains global attributes simple-parts relation: lists the parts and their attributes connects-supports relation: gives connections between

pairs of parts triples relation: specifies connections between three

parts at a time parallel relation: lists pairs of parts that are parallel perpendicular relation: lists pairs of parts that are

perpendicular TYPE: 1 for stick, 2 for plate, 3 for blob


Joke


18.3.5 Generalized Cylinder Representation

generalized cylinder: volumetric primitive defined by axis and cross-section

cross section: swept along axis, creating a solid e.g. actual cylinder: generalized cylinder whose axis is straight-

line segment and whose cross section is circle of constant radius

e.g. cone: generalized cylinder whose axis is straight-line segment and cross section is circle with radius initially zero to maximum



e.g. rectangular solid: generalized cylinder whose axis is straight line segment and cross section is constant rectangle

e.g. torus: generalized cylinder whose axis is circle and whose cross section is constant circle

generalized cylinder representation: uses generalized cylinders as primitives

torus



surface-edge-vertex model: very precise

sticks-plates-and-blobs model: very rough

generalized cylinder model: somewhere in between



person: modeled roughly as cylinders for head, torso, arms, legs

dotted lines: axes of cylinders


Joke


18.3.6 Super-quadric Representation

Super-quadrics: lumps of clay deformable and can be glued into object models

Super-quadric models: mainly used with range data


18.3.6 Super-quadric Representation

Super-quadrics are a flexible family of 3-dimensional parametric objects, useful for geometric modeling.

By adjusting a relatively few number of parameters, a large variety of shapes may be obtained.


Figure 18.13 Range data image of (a) a doll, (b) its super-quadric fit (c), (d) wire frame


Joke


18.3.7 Octree Representation octree encoding:

geometric modeling technique used to represent 3D objects used in computer vision, robotics, computer graphics

octree hierarchical: 8-ary tree structure

each node in octree corresponds to cubic region of universe

18.3.7 Octree Representation

DC & CV Lab.CSIE NTU


18.3.7 Octree Representation

full, empty, partial full:

if cube is completely enclosed by 3D object empty:

if cube contains no part of object partial:

if cube partly intersects object

partial: has eight children representing partition of cube into octants

labeled full or empty : no children


Joke


18.3.8 The Extended Gaussian Image

3D object: collection of surface normals, one at each point of object surface

planar surface: all points on surface map to same surface normal

convex with positive curvature everywhere: distinct surface normal everywhere

set of surface normals can be mapped to a unit sphere (Gaussian sphere) by placing tail at center head outward

Gaussian image of object: resultant set of points on Gaussian sphere



for planar objects: Gaussian image not invertible, not precise enough for use

δO: small surface patch of object δS: corresponding surface patch on Gaussian sphere Gaussian curvature K:

dS

dO

O

SK

O

0lim



(ξ,η): point on Gaussian sphere corresponding to point (u, v) on object surface

extended Gaussian image:

planar region: Gaussian curvature 0, point mass in extended Gaussian image

),(

1,

vuKG


Joke


18.3.9 View-Class Representation view classes: each representing set of viewpoints

sharing some property same object surfaces visible same line segments visible relational distances between relational structures are similar

characteristic views: sets producing topologically isomorphic line drawings


18.3.9 View-Class Representation three view classes of cube producing topologically

isomorphic line drawings


18.3.9 View-Class Representation aspect graph of object: graph structure where

1. each node represents topologically distinct view of object

2. a node for each such view of object

3. each arc represents a visual event at transition

4. there is an arc for each such transition


Joke


18.4 General Frameworks for Matching

matching: finding correspondence between two entities

consistent labeling procedures: examples of matching algorithms


18.4 General Frameworks for Matching

18.4.1 Relational-Distance Approach to Matching 18.4.2 Ordered Structural Matching 18.4.3 Hypothesizing and Testing with Viewpoint

Consistency Constraint 18.4.4 View-Class Matching 18.4.5 Affine-Invariant Matching


18.4.1 Relational-Distance Approach to Matching

relational distance: compares two structures and determines similarity

Relational-Distance Definition Dx : relational description

Dx = {R1 , … , RI} : sequence of relations X : set of parts of entity being described Ri : relation indicating various relationships among parts

DA : relational description with part set A

DB : relational description with part set B



assumption: |A| = |B|, otherwise add dummy parts to smaller set f: any one-one, onto mapping from A to B N: positive integer

composition R 。 F of relation with function f:

NnbafwithRaaBbbfR nnNN

N ,...,1, ,...,|,..., 11

NAR



f: maps parts from set A to parts from set B structural error of f for Ith pair of corresponding relations

in DA, DB :

total error of f with respect to DA, DB:

relational distance GD(DA, DB) between DA, DB:

||||)( 1iiii

iS RfSSfRfE

I

i

iS fEfE

1

fEDDGD

ontoBAf

lBA

:

1min,

1

1

,...,

,...,

A I

B I

D R R

D S S



best mapping from DA to DB : mapping f that minimizes total error


best mapping from to is

for this mapping:

4,3,2,1A dcbaB ,,, dfcfbfaf 4,3,2,1


0

31213121

31213121

1 32

3121323121

3121323121

}332211{

321M 321M

11

21

)},)(,{()},)(,{(

)},)(,{(f')}',')(',{(RfS

)'',(

')}',')(',{(')}',')(',')(',{(

')}',')(',{(f)},)(,)(,{(SfR

')', f()', f()f(

'', ', , ,

1


1

56561516454

32413121

6532312161516454

32413121

6532312161516454

1 54

6532312161516454

32413121

6532312161516454

32413121

}665544332211{

654321M 4321M

1****************

1

**

****************

********

****************

43

}),)(,)(,)(,)(,{(

'')}'','')('','')('','')('',{(

)},)(,)(,)(,)(,)(,)(,)(,{(

'')}'','')('','')('','')('',{(

f)},)(,)(,)(,)(,)(,)(,)(,{(

RfS

),(

)},)(,)(,)(,)(,)(,)(,)(,{(

)},)(,)(,)(,{(

)},)(,)(,)(,)(,)(,)(,)(,{(

f'')}'','')('','')('','')('',{(SfR

''), f(''), f(''), f(''), f(''), f('')f(

, , , , , '''', '', '',

''''''''''''''''''''

''''''''''''''''''''''''''''''''

******

******



Relational Distance as a Metric relational distance:

used to determine similarity of unknown object to an object model can also be used to compare object models to grouping models in

a large database

f relational isomorphism: if f one-one, onto from A to B and E(f) = 0

f: A → B relational isomorphism: DA, DB isomorphic

GD: relational-distance measure



DA, DB , DC : metric property of GD:

BCCABA

ABBA

BABA

,DDGD,DDGD,DDGD

,DDGD,DDGD

,DD,DDGD

.3

.2

isomorphic 0 .1



Attributed Relational Descriptions and Relational Distance

extend relational description and relational distance to include properties of parts properties of the whole properties of these relationships


Joke


18.4.2 Ordered Structural Matching

definition of ordering on primitives: greatly reduces complexity of search


Joke


18.4.3 Hypothesizing and Testing with Viewpoint Consistency Constraint

viewpoint consistency constraint: The locations of all projected model features in an image must

be consistent with projection from a single viewpoint.


Joke


18.4.4 View-Class Matching if 3D object represented by view-class model, matching

divided into 2 stages:

1. determining view class of object

2. determining precise viewpoint within that view class


18.4.4 View-Class Matching relational pyramid:

hierarchical relational structure to represent view class

Level-1 primitives: straight- and curved-line segments

Level-2 relations: junctions and loops

Level-3 relations: adjacency, collinearity, junction parallelness, loop-inside-loop


18.4.4 View-Class Matching Pose Determination within View Class

relational pyramid: hierarchical, relational structure to constrain matching


Joke


18.4.5 Affine-Invariant Matching

set of interest points lying in z = z0 plane rotation matrix relating model reference frame to camera

reference frame:

translation of object reference frame to camera reference frame:

M

mmm zyxM 10,,

333231

232221

131211

rrr

rrr

rrr

3

2

1

t

t

t


18.4.5 Affine-Invariant Matching f: distance between image plane and center of

perspectivity : observed image data points by perspective

projection:

when translation t3 in z-direction large compared with r31xm + r32ym :

2

1

2221

1211

b

b

y

x

rr

rr

v

u

m

m

m

m

N

nnn vuO 1,

30333231

10131211

tzryrxr

tzryrxrfu

mm

mmn

30333231

20232221

tzryrxr

tzryrxrfv

mm

mmn


18.4.5 Affine-Invariant Matching A : 2 x 2 (scaling, rotation, skewing) matrix b : 2D (translation) vector affine 2D correspondence: Aw + b


18.4.5 Affine-Invariant Matching necessary and sufficient to define plane uniquely:

3 noncollinear points


18.4.5 Affine-Invariant Matching The Hummel-Wolfson-Lamdan Matching Algorithm to match noncollinear triplets in model interest points

with scene: Step 1: preprocessing: convert model interest points into affine-

invariant model Step 2: recognition: match model against image using affine

representation


18.4.5 Affine-Invariant Matching Shortcomings of the Affine-Invariant Matching Technique affine-invariant matching technique:

mathematically sound in noiseless case

shortcomings of affine-invariant matching in practice: 1. if three noncollinear points not numerically stable, points not

reliable 2. coordinates of detected interest points: noisy in real image 3. partial object symmetries may cause wrong matching


Joke


18.5 Model Database Organization organize database of models: to allow rapid access to

most likely candidate group similar relational models into clusters and choose

representative arrows: indicate mapping from parts of object 2 to parts

of other objects


END

computer and robot vision ii

Documents