aularacornerslucas.ppt

Extração de Características

cap 4 – Trucco e Verri

Características de uma imagem

• Globais: histograma, conteúdo de freqüências, etc...

• Locais: regiões com determinada propriedade, arestas, cantos, curvas, etc...

Arestas e cantos

• Locais de mudanças significativas na intensidade da imagem

Edgedels = edge elements

Tipos de arestasdegrau(step) rampa(ramp)

cume(roof) impulso(spike)

Gráfico sem e com ruído

Derivadas e arestas

f(x) f(x)+n(x) | f'(x)+n'(x) | f"(x)+n"(x)

Série de Taylor

)()(2

)()()()()( 3"

2' xOxf

xxfxxfxxf

iiii ffff "'1 2

1

Com x=1, f(x)=fi e f(x+x)=fi+1

Com x=-1, f(x)=fi e f(x+x)=fi-1

iiii ffff "'1 2

1

(a)

(b)

Aproximações para derivadas

(a-b) 2/)( 11'

iii fff 2/)( 11'

iii fff

(a+b) )2( 11"

iiii ffff )2( 11"

iiii ffff

f(x)

x

fi-1

fi fi+1

i+1ii-1

Em 2D

y

fx

f

yxf ),(

Gradiente

Laplaciano

2

2

2

22 ),(

y

f

x

fyxf

x

yxfyxf

x

yxf mnmn

,,, 1

x

yxfyxf

y

yxf mnmn

,,, 1

11

1

1

Convolution Kernels

141

4204

141

Laplaciano

)2( 11"

iiii ffff

•Sometimes we are interested only in changing magnitude without regard to the changing orientation.

•A linear differential operator called the Laplacian may be used.

•The Laplacian has the same properties in all directions and is therefore invariant to rotation in the image.

http://www.cee.hw.ac.uk/hipr/html/log.html

http://ct.radiology.uiowa.edu/~jiangm/courses/dip/html/node83.html

Finite differences

11* II x

1

1*II y

IKhurram Hassan-Shafique

Classical Operators

Prewitt’s Operator

11

11

11

1

1

11

Smooth Differentiate

101

101

101

111

000

111

111

111

Khurram Hassan-Shafique

Classical Operators

Sobel’s Operator

11

22

11

1

1

11

SmoothDifferentiate

101

202

101

121

000

121

121

121


• Sobel Edge Detector

Detecting Edges in Image

Image I

101

202

101

121

000

121

*

*

Idx

d

Idy

d

22

Idy

dI

dx

d

ThresholdEdges


Sobel Edge Detector

Idx

d

Idy

d

I


Sobel Edge Detector

I

22

I

dy

dI

dx

d

100 ThresholdKhurram Hassan-Shafique

Marr and Hildreth Edge Operator

• Smooth by Gaussian

• Use Laplacian to find derivatives

IGS * 2

22

2

2

1

yx

eG

Sy

Sx

S2

2

2

22



IGIGS ** 222

2

22

22

22

3

2 22

1

yx

eyx

G



2

22

22

22

3

2 22

1

yx

eyx

G

0.0008 0.0066 0.0215 0.031 0.0215 0.0066 0.00080.0066 0.0438 0.0982 0.108 0.0982 0.0438 0.00660.0215 0.0982 0 -0.242 0 0.0982 0.0215

0.031 0.108 -0.242 -0.7979 -0.242 0.108 0.0310.0215 0.0982 0 -0.242 0 0.0982 0.02150.0066 0.0438 0.0982 0.108 0.0982 0.0438 0.00660.0008 0.0066 0.0215 0.031 0.0215 0.0066 0.0008

X

Y



Zero CrossingsDetection

I ImageG2*

IG *2 Edge Image

IG *2 Zero Crossings


1

6

3


Quality of an Edge Detector

• Robustness to Noise

• Localization

• Too Many/Too less Responses

Poor robustness to noise Poor localization Too many responses

True Edge


Canny Edge Detector

• Criterion 1: Good Detection: The optimal detector must minimize the probability of false positives as well as false negatives.

• Criterion 2: Good Localization: The edges detected must be as close as possible to the true edges.

• Single Response Constraint: The detector must return one point only for each edge point.


Hai Tao

The result– General form of the filter (N.B. the filter is odd so h(x) = -h(-x) the following expression

is for x < 0 only)

h x e a x a x e a x a xx x( ) ( sin cos ) ( sin cos ) / 1 2 3 4 1 2

2 05220

2 91540

156939

.

.

.

a

a

a

a

1

2

3

4

1

01486768717

0 2087553476

1244653939

0 7912446531

2

.

.

.

.

Camillo J. Taylor

Approximation– Canny’s filter can be approximated by the derivative of a Gaussian

h xd

dxe

xe

x x

( ) ( )

2

2

2

222

2

Camillo J. Taylor

Derivative of GaussianCanny

Canny Edge Detector

• Convolution with derivative of Gaussian

• Non-maximum Suppression

• Hysteresis Thresholding


Algorithm Canny_Enhancer• Smooth by Gaussian

IGS * 2

22

2

2

1

yx

eG

Tyx

T

SSSy

Sx

S

22yx SSS

x

y

S

S1tan Khurram Hassan-Shafique

• Compute x and y derivatives

• Compute gradient magnitude and orientation

Canny Edge Operator

IGIGS ** T

y

G

x

GG

T

Iy

GI

x

GS

**


Canny Edge Detector

xS

yS

I


Canny Edge Detector

I

22yx SSS

25 ThresholdS


We wish to mark points along the curve where the magnitude is biggest.We can do this by looking for a maximum along a slice normal to the curve(non-maximum suppression). These points should form a curve. There arethen two algorithmic issues: at which point is the maximum, and where is thenext one?

Algorithm Non-Maximum Suppression


Non-Maximum Suppression

• Suppress the pixels in ‘Gradient Magnitude Image’ which are not local maximum

edgean tonormaldirection thealong

in of neighbors theare and Sx,yy,xy,x

otherwise0,,&

,, if,

, yxSyxS

yxSyxSyxS

yxM

yx ,

yx,

yx ,



0

12

3

41420tan41422- :3

41422tan :2

41422tan41420 :1

41420tan41420 :0

.θ.

.θ

.θ.

.θ.-

x

y

S

Sθ tan



22yx SSS M

25ThresholdM


Hysteresis Thresholding



• If the gradient at a pixel is above ‘High’, declare it an ‘edge pixel’

• If the gradient at a pixel is below ‘Low’, declare it a ‘non-edge-pixel’

• If the gradient at a pixel is between ‘Low’ and ‘High’ then declare it an ‘edge pixel’ if and only if it is connected to an ‘edge pixel’ directly or via pixels between ‘Low’ and ‘ High’



M 25ThresholdM

15

35

Low

High


Resultado de algoritmo de histerese

Subpixel Localization– One can try to further localize the position of the edge within a pixel by analyzing the response to the

edge enhancement filter

– One common approach is to fit a quadratic polynomial to the filter response in the region of a maxima and compute the true maximum.

a

bx

yyya

yyb

cbay

cbay

cy

cbxaxxy

2

);0())1()1((2

1

));1()1((2

1

)1(

;)1(

;)0(

)(

max

2

0 1-1

Derivadas direcionais

y

n

y

f

x

n

x

f

n

f

x

y

f(x,y)

y

x

n

nn

h

pfnhpf

n

pfh

)()(lim

)(0

yx ny

fn

x

f

n

f

nfn

f

y

xyx n

n

y

f

x

f

y

fx

f

nnn

f

n

f

n

f

2

Detecting corners

– If Ex and Ey denote the gradients of the intensity image, E(x,y), in the x and y directions then the behavior of the gradients in a region around a point can be obtained by considering the following matrix

2

2

yyx

yxxyx

y

x

EEE

EEEEE

E

EC

Camillo J. Taylor

Examining the matrix

– One way to decide on the presence of a corner is to look at the eigenvalues of the 2 by 2 matrix C.

• If the area is a region of constant intensity we would expect both eigenvalues to be small

• If it contains a edge we expect one large eigenvalue and one small one

• If it contains edges at two or more orientations we expect 2 large eigenvalues

Camillo J. Taylor

Finding corners

– One approach to finding corners is to find locations where the smaller eigenvalue is greater than some threshold

– We could also imagine considering the ratio of the two eigenvalues

Computing Image Gradients

Corner Analysis– The ellipses indicate the eignvalues and eigenvectors of the C matrices

Corner Detection

• Basic idea: Find points where two edges meet—i.e., high gradient in two directions

• “Cornerness” is undefined at a single pixel, because there’s only one gradient per point– Look at the gradient behavior over a small window

• Categories image windows based on gradient statistics– Constant: Little or no brightness change– Edge: Strong brightness change in single direction– Flow: Parallel stripes– Corner/spot: Strong brightness changes in orthogonal directions

Corner Detection: Analyzing Gradient Covariance

• Intuitively, in corner windows both Ix and Iy should be high– Can’t just set a threshold on them directly, because we want rotational invariance

• Analyze distribution of gradient components over a window to differentiate between types from previous slide:

• The two eigenvectors and eigenvalues ¸1, ¸2 of C (Matlab: eig(C)) encode the predominant directions and magnitudes of the gradient, respectively, within the window

• Corners are thus where min(¸1, ¸2) is over a threshold courtesy of Wolfram

Contents

• Harris Corner Detector

– Description

– Analysis

• Detectors

– Rotation invariant

– Scale invariant

– Affine invariant

• Descriptors


– Scale invariant


Harris Detector: Mathematics

2,

( , ) ( , ) ( , ) ( , )x y

E u v w x y I x u y v I x y

Change of intensity for the shift [u,v]:

IntensityShifted intensity

Window function

orWindow function w(x,y) =

Gaussian1 in window, 0 outside


( , ) ,u

E u v u v Mv

For small shifts [u,v] we have a bilinear approximation:

2

2,

( , ) x x y

x y x y y

I I IM w x y

I I I

where M is a 22 matrix computed from image derivatives:

Harris Detector: Geometric Interpretation

( , ) ,u

E u v u v Mv

constv

uvu

v

uRRvuE T

'

'

0

0''

0

0

2

1

2

1

1, 2 – eigenvalues of M

(max)-1/2

(min)-1/2

constvu 22

21 )'()'(


1

2

“Corner”1 and 2 are large,

1 ~ 2;

E increases in all directions

1 and 2 are small;

E is almost constant in all directions

“Edge” 1 >> 2

“Edge” 2 >> 1

“Flat” region

Classification of image points using eigenvalues of M:


Measure of corner response:

2det traceR M k M

1 2

1 2

det

trace

M

M

(k – empirical constant, k = 0.04-0.06)


2

• R depends only on eigenvalues of M

• R is large for a corner

• R is negative with large magnitude for an edge

• |R| is small for a flat region

1

“Corner”

“Edge”

“Edge”

“Flat”

R > 0

R < 0

R < 0|R| small

Algoritmo

• Comparação dos gráficos

1

“Corner”

“Edge”

“Edge”

“Flat”

R > 0

R < 0

R < 0|R| small

2

2det traceR M k M

Harris Detector

• The Algorithm:– Find points with large corner response function R (R >

threshold)

– Take the points of local maxima of R

Harris Detector: Workflow


Compute corner response R


Find points with large corner response: R>threshold


Take only the points of local maxima of R

Example: Gradient Covariances

Full imageDetail of image with gradient covar-

iance ellipses for 3 x 3 windows

from Forsyth & Ponce

Corners are where both eigenvalues are big

Example: Corner Detection (for camera calibration)

courtesy of B. Wilburn

Example: Corner Detection

courtesy of S. Smith

SUSAN corners

Harris Detector: Summary

• Average intensity change in direction [u,v] can be expressed as a bilinear form:

• Describe a point in terms of eigenvalues of M:measure of corner response

• A good (corner) point should have a large intensity change in all directions, i.e. R should be large positive

( , ) ,u

E u v u v Mv

2

1 2 1 2R k

Contents


– Description

– Analysis

• Detectors


– Scale invariant


• Descriptors


– Scale invariant


Tracking: compression of video information

• Harris response (uses criss-cross gradients)• Dinosaur tracking (using features)• Dinosaur Motion tracking (using correlation)• Final Tracking (superimposed)Courtesy: (http://www.toulouse.ca/index.php4?/CamTracker/index.php4?/CamTracker/FeatureTracking.html)

This figure displays results of feature detection over the dinosaur test sequence with the algorithm set to extract the 6 most "interesting" features at every image frame.

It is interesting to note that although no attempt to extract frame-to-frame feature correspondences was made, the algorithm still extracts the same set of features at every frame.

This will be useful very much in feature tracking.

http://www.toulouse.ca/index.php4?/CamTracker/index.php4?/CamTracker/FeatureTracking.html

One More..

• Office sequence

• Office Tracking

Harris Detector: Some Properties

• Rotation invariance

Ellipse rotates but its shape (i.e. eigenvalues) remains the same

Corner response R is invariant to image rotation


• Partial invariance to affine intensity change

Only derivatives are used => invariance to intensity shift I I + b

Intensity scale: I a I

R

x (image coordinate)

threshold

R

x (image coordinate)


• But: non-invariant to image scale!

All points will be classified as edges

Corner !

Harris Detector: Some Properties• Quality of Harris detector for different scale changes-- Correspondences calculated using distance (and threshold)

-- Improved Harris is proposed by Schmid et al

-- repeatability rate is defined as the number of points

repeated between two images w.r.t the total number of

detected points. Repeatability rate:

# correspondences# possible correspondences

C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Imp.Harris uses derivative of Gaussianinstead of standard template used byHarris et al.

Contents


– Description

– Analysis

• Detectors


– Scale invariant


• Descriptors


– Scale invariant


We want to:

detect the same interest points regardless of image changes

Models of Image Change

• Geometry– Rotation

– Similarity (rotation + uniform scale)

– Affine (scale dependent on direction)valid for: orthographic camera, locally planar object

• Photometry– Affine intensity change (I a I + b)

Contents


– Description

– Analysis

• Detectors


– Scale invariant


• Descriptors


– Scale invariant


Rotation Invariant Detection


C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000

Contents


– Description

– Analysis

• Detectors


– Scale invariant


• Descriptors


– Scale invariant


Scale Invariant Detection• Consider regions (e.g. circles) of different sizes around a point

• Regions of corresponding sizes (at different scales) will look the same in both images

Fine/Low Coarse/High

Scale Invariant Detection• The problem: how do we choose corresponding circles independently

in each image?

Scale Invariant Detection

• Solution:

– Design a function on the region (circle), which is “scale invariant” (the same for corresponding regions, even if they are at different scales)

Example: average intensity. For corresponding regions (even of different sizes) it will be the same.

scale = 1/2

– For a point in one image, we can consider it as a function of region size (circle radius)

f

region size

Image 1 f

region size

Image 2


• Common approach:

scale = 1/2

f

region size/scale

Image 1 f

region size/scale

Image 2

Take a local maximum of this function

Observation: region size (scale), for which the maximum is achieved, should be invariant to image scale.

s1 s2

Important: this scale invariant region size is found in each image independently!

Max. is called characteristic scale

Characteristic Scale

• The ratio of the scales, at which the extrema were found for corresponding points in two rescaled images, is equal to the scale factor between the images.

• Characteristic Scale: Given a point in an image, compute the function responses for several factors sn The characteristic scale is the local max. of the function (can be more than one).

• Easy to look for zero-crossings of 2nd derivative than maxima.

scale = 1/2

f

region size/scale

Image 1 f

region size/scale

Image 2

s1 s2

Max. is called characteristic scale


• A “good” function for scale detection: has one stable sharp peak

f

region size

bad

f

region size

bad

f

region size

Good !

• For usual images: a good function would be the one with contrast (sharp local intensity change)


• Functions for determining scale

2 2

21 22

( , , )x y

G x y e

2 ( , , ) ( , , )xx yyL G x y G x y

( , , ) ( , , )DoG G x y k G x y

Kernel Imagef Kernels:

where Gaussian

Note: both kernels are invariant to scale and rotation

(Laplacian)

(Difference of Gaussians)

Build Scale-Space Pyramid

• All scales must be examined to identify scale-invariant features

• An efficient function is to compute the Difference of Gaussian (DOG) pyramid (Burt & Adelson, 1983) (or Laplacian)

B l u r

R e s a m p l e

S u b t r a c t

B l u r

R e s a m p l e

S u b t r a c t

B lu r

R esam p le

S u b trac t

Key point localization

• Detect maxima and minima of difference-of-Gaussian in scale space

B l u r

R e s a m p l e

S u b t r a c t

Scale Invariant Detectors• Harris-Laplacian1

Find local maximum of:– Harris corner detector in space

(image coordinates)– Laplacian in scale

1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 20012 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004

scale

x

y

Harris

L

apla

cian

• SIFT (Lowe)2

Find local maximum of:– Difference of Gaussians in

space and scale

scale

x

y

DoG

D

oG

Normal, Gaussian..

A normal distribution in a variate with mean and variance 2 is a statistic distribution with probability function

http://mathworld.wolfram.com/Variate.html

http://mathworld.wolfram.com/Mean.html

http://mathworld.wolfram.com/Variance.html

http://mathworld.wolfram.com/ProbabilityFunction.html

Harris-Laplacian

• Existing methods search for maxima in the 3D representation of an image (x,y,scale). A feature point represents a local maxima in the surrounding 3D cube and its value is higher than a threshold.

• THIS (Harris-Laplacian) method uses Harris function first, then selects points for which Laplacian attains maximum over scales.

• First, prepare scale-space representation for the Harris function. At each level, detect interest points as local maxima in the image plane (of that scale) – do this by comparing 8-neighborhood. (different from plain Harris corner detection)

• Second, use Laplacian to judge if each of the candidate points found on different levels, if it forms a maximum in the scale direction. (check with n-1 and n+1)

Scale Invariant Detectors

K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

• Experimental evaluation of detectors w.r.t. scale change

Repeatability rate:

# correspondences# possible correspondences

(points present)

Scale Invariant Detection: Summary

• Given: two images of the same scene with a large scale difference between them

• Goal: find the same interest points independently in each image• Solution: search for maxima of suitable functions in scale and in space

(over the image)

Methods:

1. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris’ measure of corner response over the image

2. SIFT [Lowe]: maximize Difference of Gaussians over scale and space

Contents


– Description

– Analysis

• Detectors


– Scale invariant

– Affine invariant (maybe later)

• Descriptors


– Scale invariant


Affine Invariant Detection

• Above we considered:Similarity transform (rotation + uniform scale)

• Now we go on to:Affine transform (rotation + non-uniform scale)


• Take a local intensity extremum as initial point

• Go along every ray starting from this point and stop when extremum of function f is reached

T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

f

points along the ray

• We will obtain approximately corresponding regions

Remark: we search for scale in every direction


• Algorithm summary (detection of affine invariant region):– Start from a local intensity extremum point– Go in every direction until the point of extremum of some function f– Curve connecting the points is the region boundary– Compute geometric moments of orders up to 2 for this region– Replace the region with ellipse

T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000.

Affine Invariant Detection : Summary

• Under affine transformation, we do not know in advance shapes of the corresponding regions

• Ellipse given by geometric covariance matrix of a region robustly approximates this region

• For corresponding regions ellipses also correspond

Methods:

1. Search for extremum along rays [Tuytelaars, Van Gool]:

2. Maximally Stable Extremal Regions [Matas et.al.]

SIFT

• Introdução

• Motivação

• Visão Geral do Algoritmo

Introdução

• O que é o SIFT?– Scale Invariant Feature Transform

David Lowe

Motivação

• Como ele ajuda em Markless?– Keypoints – Detecção de Texturas

by lowe

Visão Geral do Algoritmo

• Detecção dos máximos locais no espaço de escala– Usando a função diferença de Gaussianos (DoG)

• Localização dos Keypoint– Calculo do subpixel do keypoint

• Calculo da Orientação– 1 ou mais por Keypoint

• Keypoint descriptor– Apartir do gradiente da imagem

Detecção dos máximos

• Construção da piramide de DoG

by Brandon Green

Detecção dos máximos• Usa o DoG para calcular os keypoints

),(),,(),,( yxIyxGyxL (def. espaço de Escala)

Detecção dos máximos

• Procura nos 26 vizinhos o máximo local

Filtrando os Keypoints

• Eliminação dos máximos que tem pouco contraste com sua vizinhança

• Eliminação dos pontos fracos mal localizados (sobre linhas)

1. Imagem inicial2. Maximos3. Eliminando por contraste4. Eliminando por localização ruim

Cálculo da Orientação

• Cálculos dos Keypoints relativos a orientação.

• Permite a invariança na rotação

• Pre-calculados

Keypoint descriptor

• Calculo referente a orientação

• Histograma com 8 classes

• Melhor descritor 4x4x8 = 128 classes

Keypoint matching

• Calculo dos pares usando distancia Euclidiana– Brute force

– kd – tree

– Best-Bin-First

• Calculo da transformação para alinhamento das imagens– RANSAC

– Hough TransformNormalmente uma imagem tem ~2,000 keypoints

Resultados

aularacornerslucas.ppt

Technology