4. pattern recognition - yonsei university pattern... · 2014-12-29 · learning and adaptation [2]...

E-mail: [email protected]://web.yonsei.ac.kr/hgjung

4. Pattern 4. Pattern RecognitionRecognition


▷▷ Introduction to Pattern Recognition SystemIntroduction to Pattern Recognition System

▷▷ Feature Extraction Feature Extraction 효율화효율화: : HaarHaar--like featurelike feature와와 Integral ImageIntegral Image

▷▷ Dimension Reduction: PCADimension Reduction: PCA

▷▷ Bayesian Decision TheoryBayesian Decision Theory

▷▷ Bayesian Discriminant Function for Normal DensityBayesian Discriminant Function for Normal Density

▷▷ Linear Discriminant AnalysisLinear Discriminant Analysis

▷▷ Linear Discriminant FunctionsLinear Discriminant Functions

▷▷ Support Vector MachineSupport Vector Machine

▷▷ k Nearest Neighbork Nearest Neighbor

▷▷ Statistical ClusteringStatistical Clustering


IntroductionIntroduction


Machine Perception [2]Machine Perception [2]

• Build a machine that can recognize patterns:

– Speech recognition

– Fingerprint identification

– OCR (Optical Character Recognition)

– DNA sequence identification


Components of Pattern Classification System [6]Components of Pattern Classification System [6]


Types of Prediction Problems [6]Types of Prediction Problems [6]


Feature and Pattern [6]Feature and Pattern [6]


Classifier [6]Classifier [6]


Pattern Recognition Approaches [6]Pattern Recognition Approaches [6]



(Example)“Sorting incoming Fish on a conveyor according to species using optical sensing”

Sea bassSpecies

Salmon



Problem Analysis: set up a camera and take some sample images to extract features

Preprocessing: use a segmentation operation to isolate fishes from one another and from the background

Feature extraction: information from a single fish is sent to a feature extractor whose purpose is to reduce the data by measuring certain features

The features are passed to a classifier


Feature Selection [2]Feature Selection [2]

The length of the fish as a possible feature for discrimination

The length is a poor feature alone!



The lightness of the fish as a possible feature for discrimination



• Adopt the lightness and add the width of the fish

Fish xT = [x1, x2]

Lightness Width


Generalization [2]Generalization [2]

The central aim of designing a classifier is to correctly classify novel inputnovel input.


Generalization [3]Generalization [3]

Polynomial Curve FittingPolynomial Curve Fitting


Generalization: Model Selection [3]Generalization: Model Selection [3]


0th Order Polynomial 1st Order Polynomial

3rd Order Polynomial 9th Order Polynomial


Generalization: Model Selection [3]Generalization: Model Selection [3]

Polynomial Curve FittingPolynomial Curve Fitting, Over, Over--fittingfitting

Root‐Mean‐Square (RMS) Error:


Generalization: Sample Size [3]Generalization: Sample Size [3]


9th Order Polynomial N=15


Generalization: Sample Size [3]Generalization: Sample Size [3]


9th Order Polynomial N=100


Generalization: Regularization [3]Generalization: Regularization [3]


Regularization: Penalize large coefficient values


Learning and Adaptation [2]Learning and Adaptation [2]

• Supervised Learning: a teacher provides a category label or cost for each

pattern in a training set, and seeks to reduce the sum of the costs for

these patterns.

• Unsupervised Learning: there is no explicit teacher, and the system forms

clusters or “natural grouping” of the input patterns.

• Reinforcement Learning: no desired category signal is given; instead, the

only teaching is that the tentative category is right or wrong.


Linear Discriminant Functions [6]Linear Discriminant Functions [6]


Feature Extraction Feature Extraction 효율화효율화: : HaarHaar--like featurelike feature와와

Integral ImageIntegral Image


HaarHaar--like Feature [7]like Feature [7]

The simple features used are reminiscent of Haar basis functions which have been used by Papageorgiou et al. (1998).Three kinds of features: two-rectangle feature, three-rectangle feature, and four-rectangle featureGiven that the base resolution of the detector is 24x24, the exhaustive set of rectangle feature is quite large, 160,000.


HaarHaar--like Feature: Integral Image [7]like Feature: Integral Image [7]

Rectangle features can be computed very rapidly using an intermediate representation for the image which we call the integral image.The integral image at location x,y contains the sum of the pixels above and to the left of x, y, inclusive:

where ii (x, y) is the integral image and i (x, y) is the original image (see Fig. 2). Using the following pair of recurrences:

(where s(x, y) is the cumulative row sum, s(x,−1) =0, and ii (−1, y) = 0) the integral image can be computed in one pass over the original image.


HaarHaar--like Feature: Integral Image [7]like Feature: Integral Image [7]

Using the integral image any rectangular sum can be computed in four array references (see Fig. 3).

Our hypothesis, which is borne out by experiment, is that a very small number of these features can be combined to form an effective classifier. The main challenge is to find these features.


Dimension Dimension Reduction: PCAReduction: PCA


Abstract [1]Abstract [1]

Principal component analysis (PCA) is a technique that is useful for the compression and classification of data. The purpose is to reduce the dimensionality of a data set (sample) by finding a new set of variables, smaller than the original set of variables, that nonetheless retains most of the sample's information.

By information we mean the variation present in the sample, given by the correlations between the original variables. The new variables, called principal components (PCs), are uncorrelated, and are ordered by the fraction of the total information each retains.


Geometric Picture of Principal Components [1]Geometric Picture of Principal Components [1]

A sample of n observations in the 2-D space

GoalGoal: to account for the variation in a sample in as few variables as possible, to some accuracy


Geometric Picture of Principal Components [1]Geometric Picture of Principal Components [1]

• the 1st PC is a minimum distance fit to a line in X space• the 2nd PC is a minimum distance fit to a line in the plane perpendicular to the 1st PC

PCs are a series of linear least squares fits to a sample, each orthogonal to all the previous.


Usage of PCA: Data Compression [1]Usage of PCA: Data Compression [1]

Because the kth PC retains the kth greatest fraction of the variationwe can approximate each observation by truncating the sum at the first m < p PCs


Usage of PCA: Data Compression [1]Usage of PCA: Data Compression [1]

Reduce the dimensionality of the data

from p to m < p by approximating

where is the n x m portion of

and is the p x m portion of

n: sample number


Let X be a d-dimensional random vector expressed as column vector.

Without loss of generality, assume X has zero mean. We want to find

a orthonormal transformation matrix P such that

with the constraint that

is a diagonal matrix and

PX is a random vector with all its distinct components pairwise uncorrelated.

By substitution, and matrix algebra, we obtain:

Derivation of PCA using the Covariance Method [8]Derivation of PCA using the Covariance Method [8]


We now have:

Rewrite P as d column vectors, so

and as:

Substituting into equation above, we obtain:

Notice that in ,

Pi is an eigenvector of the covariance matrix of X. Therefore, by finding the

eigenvectors of the covariance matrix of X, we find a projection matrix P

that satisfies the original constraints.

Derivation of PCA using the Covariance Method [8]Derivation of PCA using the Covariance Method [8]


Bayesian Decision Bayesian Decision TheoryTheory


State of Nature [2]State of Nature [2]

We let denote the state of nature, with = 1 for sea bass and = 2 for

salmon.

Because the state of nature is so unpredictable, we consider to be a

variable that must be described probabilistically.

P(1) = P(2) (uniform priors)

P(1) + P( 2) = 1 (exclusivity and exhaustivity)

More generally, we assume that there is some a priori probability (or simply

prior) P(1) that the next fish is sea bass, and some prior probability P(2)

that it is salmon.

P(1) + P( 2) = 1 (exclusivity and exhaustivity)

Decision rule with only the prior information

Decide 1 if P(1) > P(2) otherwise decide 2


ClassClass--Conditional Probability Density [2]Conditional Probability Density [2]

In most circumstances we are not asked to make decisions with so little

information. In our example, we might for instance use a lightness

measurement x to improve our classifier.

We consider x to be a continuous random variable whose distribution

depends on the state of nature and is expressed as p(x|). This is the

class-conditional probability density function, the probability density

function for x given that the state of nature is .

Hypothetical class-conditional probability density functions show the probability density of measuring a particular feature value x given the pattern is in category ωi.


Suppose that we know both the prior probabilities P(j) and the conditional

densities p(x|j) for j=1, 2.

Suppose further that we measure the lightness of a fish and discover that

its value is x.

How does this measurement influence our attitude concerning the How does this measurement influence our attitude concerning the

true state of nature true state of nature –– that is, the category of the fish?that is, the category of the fish?

Posterior, likelihood, evidence [2]Posterior, likelihood, evidence [2]



Bayes formula:

Where in case of two categories

Then,

2j

1jjj )(P)|x(P)x(P

likelihood priorposteriorevidence

( | ) ( )( | )

( )j j

j

P x PP x

P x

2

1

( | ) ( ) ( | ) ( )( | )

( ) ( | ) ( )

j j j jj j

j jj

P x P P x PP x

P x P x P



Posterior probabilities for the particular priors P(ω1) = 2/3 and P(ω2)= 1/3for the class-conditional probability densities shown in Fig. 2.1.

Thus in this case, given that a pattern is measured to have feature value x= 14, the probability it is in category ω2 is roughly 0.08, and that it is in ω1

is 0.92.At every x, the posteriors sum to 1.0.


Decision given the Posterior Probabilities [2]Decision given the Posterior Probabilities [2]

x is an observation for which:

if P(1 | x) > P(2 | x) True state of nature = 1

if P(1 | x) < P(2 | x) True state of nature = 2

Therefore:

Whenever we observe a particular x, the probability of error is :

P(error | x) = P(1 | x) if we decide 2

P(error | x) = P(2 | x) if we decide 1

Decide 1 if P(1 | x) > P(2 | x);otherwise decide 2

Therefore:P(error | x) = min [P(1 | x), P(2 | x)]

(Bayes decision)


Bayesian Decision Theory : Risk Minimization [2]Bayesian Decision Theory : Risk Minimization [2]

Generalization of the preceding ideas

- Use of more than one feature

- Use more than two states of nature

-- Allowing actions and not only decide on the state of natureAllowing actions and not only decide on the state of nature

-- Introduce of a loss function which is more general than the proIntroduce of a loss function which is more general than the probability bability

of errorof error

Feature vector & feature space

Feature vector x is in a d-dimensional Euclidean space Rd, called the

feature space.


Risk Minimization: Loss Function [2]Risk Minimization: Loss Function [2]

Formally, the loss function states how costly each action taken is, and is

used to convert a probability determination into a decision.

Let {1, 2,…, c} be the set of c states of nature (or “categories”)

Let {1, 2,…, a} be the set of possible actions

Let (i | j) be the loss incurred for taking action i when the state of

nature is j

Overall riskR = Sum of all R(i | x) for i = 1,…,a

Minimizing R Minimizing R(i | x) for i = 1,…, a

for i = 1,…,a

cj

1jjjii )x|(P)|()x|(R

|R R p d x x x x

Conditional risk


Risk Minimization [2]Risk Minimization [2]

Two Category Classification

1 : deciding 1

2 : deciding 2

ij = (i | j)

loss incurred for deciding i when the true state of nature is j

Conditional risk:

R(1 | x) = 11P(1 | x) + 12P(2 | x)

R(2 | x) = 21P(1 | x) + 22P(2 | x)



Our rule is the following:

if R(1 | x) < R(2 | x)

action 1: “decide 1” is taken

This results in the equivalent rule :

decide 1 if:

(21- 11) P(x | 1) P(1) > (12- 22) P(x | 2) P(2)

and decide 2 otherwise




Likelihood ratioLikelihood ratio:

The preceding rule is equivalent to the following rule:

Then take action 1 (decide 1)

Otherwise take action 2 (decide 2)

Optimal decision propertyOptimal decision property

“If the likelihood ratio exceeds a threshold value independent of the input

pattern x, we can take optimal actions”

)(P)(P.)|x(P

)|x(P if1

2

1121

2212

2

1



Minimum Error Rate Classification [2]Minimum Error Rate Classification [2]

Actions are decisions on classes

If action i is taken and the true state of nature is j then:

the decision is correct if i = j and in error if i j

Seek a decision rule that minimizes the probability of error which is the

error rate

Introduction of the zero-one loss function:

Therefore, the conditional risk is:

“The risk corresponding to this loss function is the average probability error”

c,...,1j,i ji 1ji 0

),( ji

1jij

cj

1jjjii

)x|(P1)x|(P

)x|(P)|()x|(R


Bayesian Decision Theory : Continuous Features [2]Bayesian Decision Theory : Continuous Features [2]

Generalization of the preceding ideas

-- Use of more than one featureUse of more than one feature

-- Use more than two states of natureUse more than two states of nature

- Allowing actions and not only decide on the state of nature

- Introduce a loss of function which is more general than the

probability of error


Classifier, Discriminant Functions, and Decision Surface [2]Classifier, Discriminant Functions, and Decision Surface [2]

Set of discriminant functions gi(x), i = 1,…, c

The classifier assigns a feature vector x to class i

if: gi(x) > gj(x) j i

The functional structure of a general statistical pattern classifier which includes dinputs and c discriminant functions gi (x). A subsequent step determines which of the discriminant values is the maximum, and categorizes the input pattern accordingly. The arrows show the direction of the flow of information, though frequently the arrows are omitted when the direction of flow is self-evident.


The MultiThe Multi--category casecategory case

Let gi(x) = - R(i | x)(max. discriminant corresponds to min. risk!)

For the minimum error rate, we take gi(x) = P(i | x)

(max. discrimination corresponds to max. posterior!)gi(x) P(x | i) P(i)

gi(x) = ln P(x | i) + ln P(i)(ln: natural logarithm!)

Feature space divided into c decision regions

if if ggii(x(x) > ) > ggjj(x(x) ) j j i then x is in i then x is in RRii

((RRii means assign means assign xx to to ii))



The twoThe two--category casecategory case

A classifier is a “dichotomizer” that has two discriminant functions g1

and g2

Let g(x) g1(x) – g2(x)

Decide 1 if g(x) > 0 ;

Otherwise decide 2

The computation of g(x)

)()(ln

)|()|(ln

)|()|()(

2

1

2

1

21

PP

xPxP

xPxPxg



In this two-dimensional two-category classifier, the probability densities are Gaussian, the decision boundary consists of two hyperbolas, and thus the decision region R2 is not simply connected. The ellipses mark where the density is 1/e times that at the peak of the distribution.


The twoThe two--category casecategory case


Bayesian Bayesian Discriminant Discriminant

Function for Normal Function for Normal DensityDensity


The Normal Density [2]The Normal Density [2]

Univariate density

Density which is analytically tractable

Continuous density

A lot of processes are asymptotically Gaussian

Handwritten characters, speech sounds are ideal or prototype

corrupted by random process (central limit theorem)

Where:

= mean (or expected value) of x

2 = expected squared deviation or variance

,x21exp

21)x(P

2



A univariate normal distribution has roughly 95% of its area in the range|x − μ| ≤ 2σ, as shown. The peak of the distribution has value p(μ) = 1/ 2


Multivariate density

Multivariate normal density in d dimensions is:

where:

x = (x1, x2, …, xd)t (t stands for the transpose vector form)

= (1, 2, …, d)t mean vector

= d×d covariance matrix

|| and -1 are determinant and inverse respectively

)()(

21exp

)2(1)( 1

2/12/

xxxP t

d



We saw that the minimum error-rate classification can be achieved by the

discriminant function

gi(x) = ln P(x | i) + ln P(i)

Case of multivariate normal

[6]

Discriminant Function for the Normal Density [2]Discriminant Function for the Normal Density [2]

)(lnln212ln

2)()(

21)( 1

iiiit

ii Pdxxxg

11 1( ) ( ) ln ln ( )2 2

ti i i i ix x P

quadratic discriminant function


Covariance Matrix [6]Covariance Matrix [6]


Discriminant Function for the Normal Density [6]Discriminant Function for the Normal Density [6]


Discriminant Functions for the Normal Density [2]Discriminant Functions for the Normal Density [2]

If the covariance matrices for two distributions are equal and proportional to the identity matrix, then the distributions are spherical in d dimensions, and the boundary is a generalized hyperplane of d −1 dimensions, perpendicular to the line separating the means.

In these one-, two-, and three-dimensional examples, we indicate p(x|ωi ) and the boundaries for the case P(ω1) = P(ω2). In the three-dimensional case, the grid plane separates R1 from R2.



Probability densities (indicated by the surfaces in two dimensions and ellipsoidal surfaces in three dimensions) and decision regions for equal but asymmetric Gaussian distributions. The decision hyperplanes need not be perpendicular to the line connecting the means.



Arbitrary Gaussian distributions lead to Bayes decision boundaries that are general hyperquadrics. Conversely, given any hyperquadric, one can find two Gaussiandistributions whose Bayes decision boundary is that hyperquadric. These variances are indicated by the contours of constant probability density.



Arbitrary three-dimensional Gaussian distributions yield Bayes decision boundaries that are two-dimensional hyperquadrics. There are even degenerate cases in which the decision boundary is a line.


Linear Discriminant Linear Discriminant AnalysisAnalysis


LDA, TwoLDA, Two--Classes [6]Classes [6]


LDA, MultiLDA, Multi--Classes [6]Classes [6]


LDA Vs. PCA [6]LDA Vs. PCA [6]


Limitations of LDA [6]Limitations of LDA [6]


Linear Discriminant Linear Discriminant FunctionsFunctions


Linear Discriminant Functions [6]Linear Discriminant Functions [6]


Gradient Descent [6]Gradient Descent [6]


PerceptronPerceptron Learning [6]Learning [6]


Minimum Squared Error Solution [6]Minimum Squared Error Solution [6]


The PseudoThe Pseudo--Inverse Solution [6]Inverse Solution [6]


LeastLeast--MeanMean--Squares Solution [6]Squares Solution [6]


Summary: Summary: PerceptronPerceptron vs. MSE Procedures [6]vs. MSE Procedures [6]


The HoThe Ho--KashyapKashyap Procedure [6]Procedure [6]


Support Vector Support Vector MachineMachine


Optimal Separating Optimal Separating HyperplanesHyperplanes [6][6]



Distance between a plane and a point


Consider the two-dimensional optimization problem:

We can visualize contours of f given by

g(x,y) = c

f(x,y)=d

Find x and y to maximize f(x,y) subject to a constraint (shown in red) g(x,y) = c.

Lagrange Multipliers [9]Lagrange Multipliers [9]


When f(x,y) becomes maximum on the path of g(x,y)=c, the contour line for g=c meets contour lines of f tangentially. Since the gradient of a function is perpendicular to the contour lines, this is the same as saying that the gradients of f and g are parallel.

Contour map. The red line shows the constraint g(x,y) = c. The blue lines are contours of f(x,y). The point where the red line tangentially touches a blue contour is our solution.



To incorporate these conditions into one equation, we introduce an

auxiliary function

and solve



KuhnKuhn--Tucker Theorem [6]Tucker Theorem [6]


The The LagrangianLagrangian Dual Problem [6]Dual Problem [6]


Minimize (in w, b)

Subject to (for any i=1,…n)

One could be tempted to expressed the previous problem by means of non-negative Lagrange multipliers αi as

we could find the minimum by sending all αi to ∞. Nevertheless the previous constrained problem can be expressed as

This is we look for a saddle point.

Dual Problem [10]Dual Problem [10]


Support Vectors [6]Support Vectors [6]


NonNon--separable Case [6]separable Case [6]


NonNon--linear linear SVMsSVMs [6][6]


Implicit Mappings: An Example [6]Implicit Mappings: An Example [6]


Kernel Methods [6]Kernel Methods [6]


Kernel Methods [6]Kernel Methods [6]

Kernel Functions


Architecture of an SVM [6]Architecture of an SVM [6]


Case Study: XOR [6]Case Study: XOR [6]


k Nearest Neighbork Nearest Neighbor


The k Nearest Neighbor Classification Rule [6]The k Nearest Neighbor Classification Rule [6]


Statistical ClusteringStatistical Clustering


NonNon--parametric Unsupervised Learning [6]parametric Unsupervised Learning [6]


Proximity Measures [6]Proximity Measures [6]


Criterion Function for Clustering [6]Criterion Function for Clustering [6]


Cluster Validity [6]Cluster Validity [6]


Iterative Optimization [6]Iterative Optimization [6]


The kThe k--means Algorithm [6]means Algorithm [6]


The kThe k--means Algorithm [4]means Algorithm [4]


ReferencesReferences

1. Frank Masci, “An Introduction to Principal Component Analysis,”http://web.ipac.caltech.edu/staff/fmasci/home/statistics_refs/PrincipalComponentAnalysis.pdf

2. Richard O. Duda, Peter E. Hart, David G. Stork, Pattern Classification, second edition, John Wiley & Sons, Inc., 2001.

3. Christopher M. Bishop, Pattern Recognition and Machine Learning, Springer, 2007.

4. Sergios Theodoridis, Konstantinos Koutroumbas, Pattern Recognition, Academic Press, 2006.

5. Ho Gi Jung, Yun Hee Lee, Pal Joo Yoon, In Yong Hwang, and Jaihie Kim, “Sensor Fusion Based Obstacle Detection/Classification for ActivePedestrian Protection System,” Lecture Notes on Computer Science, Vol. 4292, 294-305.

6. Ricardo Gutierrez-Osuna, “Pattern Recognition, Lecture Notes,” available at http://research.cs.tamu.edu/prism/lectures.htm

7. Paul Viola, Michael Jones, “Robust real-time object detection,”International Journal of Computer Vision, 57(2), 2004, 137-154.

8. Wikipedia, “Principal component analysis,” available at

http://en.wikipedia.org/wiki/Principal_component_analysis


ReferencesReferences

9. Wikipeida, “Lagrange multipliers,”http://en.wikipedia.org/wiki/Lagrange_multipliers.

10.Wikipeida, “Support Vector Machine,”http://en.wikipedia.org/wiki/Support_vector_machine.

4. pattern recognition - yonsei university pattern... · 2014-12-29 · learning and adaptation [2]...

Documents