e igen & s ingular v alue d ecomposition قسمتی از درس ریاضی مهندسی...

EIGEN & SINGULAR VALUE DECOMPOSITION

و ) ارشد پیشرفته مهندسی ریاضی درس از قسمتیدکتری(

1-91-90

شنبه، دو

شنبه، دو

3/9/903/9/90

11

نوعی، رنجبر نوعی، دکتر رنجبر دکتر

ابزار و کنترل مهندسی ابزار گروه و کنترل مهندسی گروهدقیقدقیق

RECAP: CLUSTERING 2

Hierarchical clustering Agglomerative clustering techniques

Evaluation Term vs. document space clustering Multi-lingual docs Feature selection Labeling

شنبه، دو

شنبه، دو

3/9/903/9/90

22

EIGENVALUES & EIGENVECTORS

Eigenvectors (for a square mm matrix S)

How many eigenvalues are there at most?

شنبه، دو

شنبه، دو

3/9/903/9/90

33

only has a non-zero solution ifonly has a non-zero solution if

this is a this is a mm-th order equation in -th order equation in λλ which can which can have have at most at most mm distinct solutions distinct solutions (roots of (roots of the characteristic polynomial) – the characteristic polynomial) – can be complex even can be complex even

though though SS is real. is real.

eigenvalueeigenvalue(right) eigenvector(right) eigenvector

ExampleExample

MATRIX-VECTOR MULTIPLICATION

شنبه، دو

شنبه، دو

3/9/903/9/90

44

000

020

003

S has eigenvalues 3, 2, 0 withhas eigenvalues 3, 2, 0 withcorresponding eigenvectorscorresponding eigenvectors

0

0

1

1v

0

1

0

2v

1

0

0

3v

On each eigenvector, S acts as a multiple of the On each eigenvector, S acts as a multiple of the identityidentity

matrix: but as a different multiple on each.matrix: but as a different multiple on each.Any vector (say Any vector (say xx= ) can be viewed as a combination of= ) can be viewed as a combination of

the eigenvectors: the eigenvectors: xx = 2 = 2vv1 1 + 4+ 4vv2 2 + 6+ 6vv33

6

4

2

MATRIX VECTOR MULTIPLICATIONThus a matrix-vector multiplication such as

Sx (S, x as in the previous slide) can be rewritten in terms of the eigenvalues/vectors:

Even though x is an arbitrary vector, the action of S on x is determined by the

eigenvalues/vectors.Suggestion: the effect of “small” eigenvalues

is small .

شنبه، دو

شنبه، دو

3/9/903/9/90

55

332211321

321

642642

)642(

vvvSvSvSvSx

vvvSSx

EIGENVALUES & EIGENVECTORSشنبه،

دوشنبه،

دو3/9/903/9/90

66

0 and , 2121}2,1{}2,1{}2,1{ vvvSv

For symmetric matrices, eigenvectors for distinctFor symmetric matrices, eigenvectors for distincteigenvalues are eigenvalues are orthogonalorthogonal

TSS and 0 if ,complex for IS

All eigenvalues of a real symmetric matrix are All eigenvalues of a real symmetric matrix are realreal..

0vSv if then ,0, Swww Tn

All eigenvalues of a positive semidefinite matrixAll eigenvalues of a positive semidefinite matrix

are are non-negativenon-negative

EXAMPLE

Let

Then

The eigenvalues are 1 and 3 (nonnegative, real) .

The eigenvectors are orthogonal (and real):

شنبه، دو

شنبه، دو

3/9/903/9/90

77

21

12S

.01)2(21

12 2

IS

1

1

1

1

Real, symmetric.Real, symmetric.

Plug in these values Plug in these values and solve for and solve for eigenvectors.eigenvectors.

EIGEN/DIAGONAL DECOMPOSITION

Let be a square matrix with m linearly independent eigenvectors (a

“non-defective” matrix)

Theorem: Exists an eigen decomposition

(cf. matrix diagonalization theorem)Columns of U are eigenvectors of S

Diagonal elements of are eigenvalues of

شنبه، دو

شنبه، دو

3/9/903/9/90

88

diagonaldiagonal

UniquUnique for e for

distincdistinct t

eigen-eigen-valuesvalues

DIAGONAL DECOMPOSITION: WHY/HOW

شنبه، دو

شنبه، دو

3/9/903/9/90

99

nvvU ...1Let Let UU have the eigenvectors as columns: have the eigenvectors as columns:

n

nnnn vvvvvvSSU

............

1

1111

Then, Then, SUSU can be written can be written

And And S=US=UUU–1–1..

Thus Thus SU=USU=U, or , or UU–1–1SU=SU=

DIAGONAL DECOMPOSITION - EXAMPLE

شنبه، دو

شنبه، دو

3/9/903/9/90

1010

Recall Recall .3,1;21

1221

S

The eigenvectors and form The eigenvectors and form

1

1

1

1

11

11U

Inverting, we haveInverting, we have

2/12/1

2/12/11U

Then, Then, S=US=UUU–1 –1 ==

2/12/1

2/12/1

30

01

11

11

RecallRecallUUUU––11 =1. =1.

EXAMPLE CONTINUED

شنبه، دو

شنبه، دو

3/9/903/9/90

1111

LetLet’’s divide s divide U U (and multiply (and multiply UU–1–1)) by by 2

2/12/1

2/12/1

30

01

2/12/1

2/12/1Then, Then, S=S=

QQ (Q(Q-1-1= Q= QT T ))

Why? Stay tuned Why? Stay tuned ……

SYMMETRIC EIGEN DECOMPOSITION

If is a symmetric matrix:

Theorem: Exists a (unique) eigen decomposition

where Q is orthogonal: Q-1= QT

Columns of Q are normalized eigenvectors

Columns are orthogonal.

(everything is real)

شنبه، دو

شنبه، دو

3/9/903/9/90

1212

TQQS

EXERCISE

Examine the symmetric eigen decomposition, if any, for each of the

following matrices:

شنبه، دو

شنبه، دو

3/9/903/9/90

1313

01

10

01

10

32

21

42

22

SINGULAR VALUE DECOMPOSITION

شنبه، دو

شنبه، دو

3/9/903/9/90

1414



UNDERCONSTRAINED LEAST SQUARES

What if you have fewer data points than parameters in your function? Intuitively, can’t do standard least squares Recall that solution takes the form ATAx = ATb When A has more columns than rows,

ATA is singular: can’t take its inverse, etc.

شنبه، دو

شنبه، دو

3/9/903/9/90

1515




More subtle version: more data points than unknowns, but data poorly constrains function

Example: fitting to y=ax2+bx+c

شنبه، دو

شنبه، دو

3/9/903/9/90

1616




Problem: if problem very close to singular, round off error can have a huge effect Even on “well-determined” values!

Can detect this: Uncertainty proportional to covariance C = (ATA)-1

In other words, unstable if ATA has small values More precisely, care if xT(ATA)x is small for any x

Idea: if part of solution unstable, set answer to 0 Avoid corrupting good parts of answer

شنبه، دو

شنبه، دو

3/9/903/9/90

1717



SINGULAR VALUE DECOMPOSITION (SVD)

Handy mathematical technique that has application to many problems

Given any mn matrix A, algorithm to find matrices U, V, and S such that

A = U S VT

U is mm and OrthonormalS is mn and DiagonalV is nn and Orthonormal

شنبه، دو

شنبه، دو

3/9/903/9/90

1818



SVD

Treat as black box: code widely availableIn Matlab: [U,S,V]=svd(A,0)

شنبه، دو

شنبه، دو

3/9/903/9/90

1919

T

1 0 0

0 0

0 0 n

S

S

VA U

T

1 0 0

0 0

0 0 n

S

S

VA U



SVD

The Si are called the singular values of A

If A is singular, some of the Si will be 0

In general rank(A) = number of nonzero si

SVD is mostly unique (up to permutation of singular values, or if some Si are equal)

شنبه، دو

شنبه، دو

3/9/903/9/90

2020




TTT

T

T

T

T

T

vuvuvu

v

v

v

v

v

uuuA

333222111

5

4

3

2

1

3

2

1

321

0000

0000

0000

شنبه، دو

شنبه، دو

3/9/903/9/90

2121

THE SINGULAR VALUE DECOMPOSITION

شنبه، دو

شنبه، دو

3/9/903/9/90

2222

r = the rank of A r = the rank of A = number of linearly = number of linearly

independent independent columns/rowscolumns/rows

AA UUVVTT

m x n m x m m x n n x nm x n m x m m x n n x n

·· ··

== 00

00

THE SINGULAR VALUE DECOMPOSITION

شنبه، دو

شنبه، دو

3/9/903/9/90

2323

r = the rank of A r = the rank of A = number of linearly = number of linearly

independent independent columns/rowscolumns/rows

AA UU

m x n m x m m x n n x nm x n m x m m x n n x n

== 00

00VVTT

SVD PROPERTIES

U, V give us orthonormal bases for the subspaces of A: 1st r columns of U: Column space of A Last m - r columns of U: Left nullspace of A 1st r columns of V: Row space of A 1st n - r columns of V: Nullspace of A

IMPLICATION: Rank(A) = r

شنبه، دو

شنبه، دو

3/9/903/9/90

2424


1 1 1 2 2 2

1 2

0

T T Tr r r

r

A u v u v u v

شنبه، دو

شنبه، دو

3/9/903/9/90

2525

where where • uu11 … …uurr are the r orthonormal vectors that are basis of C(A) and are the r orthonormal vectors that are basis of C(A) and

• vv11 … …vvrr are the r orthonormal vectors that are basis of C(A are the r orthonormal vectors that are basis of C(ATT ) )

MATLAB EXAMPLE

>>A = rand(3,5)

20.20.01.06.92.

27.20.81.89.94.

60.14.35.41.41.

A

شنبه، دو

شنبه، دو

3/9/903/9/90

2626

MATLAB EXAMPLE

[ >>U,S,V = ]svd(A)

03.10.95.04.30.

00.98.06.13.16.

72.01.18.52.43.

69.00.15.51.49.

03.19.21.67.68.

0042.00

00066.0

000090.1

,

00.92.39.

47.35.81.

88.19.44.

V

SU

شنبه، دو

شنبه، دو

3/9/903/9/90

2727

SVD PROOF Any m x n matrix A has two symmetric

covariant matrices

(m x m) AAT

(n x n) ATA

شنبه، دو

شنبه، دو

3/9/903/9/90

2828

WHY IS SVD SO USEFUL?: استفاده موارد

1. Inverses2. Pseudo Inverse3. Eigen values and Eigenvectors4. Matrix equivalent using SVD as Similarity

transform5. Frobenius Norm of a Matrix6. Matrix Liklihood

شنبه، دو

شنبه، دو

3/9/903/9/90

2929



CONTINUED : استفاده موارد

7. Principal Components Analysis (PCA) on: Faces and Recognition8. Total Least Squares9. Constrained Optimization

شنبه، دو

شنبه، دو

3/9/903/9/90

3030



SVD AND INVERSES

1. Application #1: inverses A-1=(VT)-1 S-1 U-1 = V S-1 UT

Using fact that inverse = transposefor orthogonal matrices

Since S is diagonal, S-1 also diagonal with reciprocals of entries of S

شنبه، دو

شنبه، دو

3/9/903/9/90

3131



SVD AND INVERSES

A-1=(VT)-1 S-1 U-1 = V S-1 UT

This fails when some si are 0 It’s supposed to fail for singular matrix

2. Pseudo inverse: if si=0, set 1/si to 0 (!) “Closest” matrix to inverse Defined for all (even non-square, singular, etc.)

matrices Equal to (ATA)-1AT if ATA invertible

شنبه، دو

شنبه، دو

3/9/903/9/90

3232



SVD AND LEAST SQUARES

Solving Ax=b by least squares x=pseudoinverse(A) times b Compute pseudoinverse using SVD

Lets you see if data is singular Even if not singular, ratio of max to min singular

values (condition number) tells you how stable the solution will be

Set 1/si to 0 if si is small (even if not exactly 0)

شنبه، دو

شنبه، دو

3/9/903/9/90

3333



SVD AND EIGENVECTORS

Let A=USVT, and let xi be ith column of V

Consider ATA xi:

3. So elements of S are sqrt(eigenvalues) and columns of V are eigenvectors of ATA What we wanted for robust least squares fitting!

شنبه، دو

شنبه، دو

3/9/903/9/90

3434

T T T T 2 T 2 22

0 0

1

0 0

i i i i iix S S x S x S s xs

A A V U U V V V V V

T T T T 2 T 2 22

0 0

1

0 0

i i i i iix S S x S x S s xs

A A V U U V V V V V

نوعی رنجبر نوعی دکتر رنجبر ،،دکتر


SVD AND MATRIX SIMILARITY

4. One common equivalent of matrix similarity in linear system of :

Can be deduced using . This changes the linear system to

: This means of a similarity transform for the

system using SVD(A).

شنبه، دو

شنبه، دو

3/9/903/9/90

3535

X AX BU

Y CX

X AX BU

Y CX

X PXX PX

X AX BU

Y CX

X AX BU

Y CX



SVD AND MATRIX NORM

5. One common definition for the norm of a matrix is the Frobenius norm:

Frobenius norm can be computed from SVD

So changes to a matrix can be evaluated by looking at changes to singular values

شنبه، دو

شنبه، دو

3/9/903/9/90

3636

i j

ija 2

FA

i jija 2

FA

i

iw 2

FA

iiw 2

FA



SVD AND MATRIX LIKLIHOOD

6. Suppose you want to find best rank-k approximation to A

Answer: set all but the largest k singular values to zero

Can form compact representation by eliminating columns of U and V corresponding to zeroed si

شنبه، دو

شنبه، دو

3/9/903/9/90

3737



SVD AND PCA

7. Principal Components Analysis (PCA): approximating a high-dimensional data setwith a lower-dimensional subspace

شنبه، دو

شنبه، دو

3/9/903/9/90

3838

Original axesOriginal axes

****

******

**** **

**

********

**

**

****** **

**** ******

Data pointsData points

First principal componentFirst principal componentSecond principal componentSecond principal component



SVD AND PCA

Data matrix with points as rows, take SVD Subtract out mean (“whitening”)

Columns of Vk are principal components

Value of si gives importance of each component

شنبه، دو

شنبه، دو

3/9/903/9/90

3939



PHYSICAL INTERPRETATION Consider a correlation matrix, A

Error ellipse with the major axis as the larger eigenvalue and the minor axis as the smaller eigenvalue

شنبه، دو

شنبه، دو

3/9/903/9/90

4040

25.0,75.1175.

75.121

A

PHYSICAL INTERPRETATION

Orthogonal directions of greatest variance in data Projections along PC1 (Principal Component) discriminate

the data most along any one axis

شنبه، دو

شنبه، دو

3/9/903/9/90

4141

Original Variable AOriginal Variable A

Ori

gin

al V

ari

able

O

rigin

al V

ari

able

BB

PC 1PC 1PC 2PC 2

شنبه، دو

شنبه، دو

3/9/903/9/90

4242

PCA ON FACES: “EIGENFACES”

شنبه، دو

شنبه، دو

3/9/903/9/90

4343

AverageAveragefaceface

First principal componentFirst principal component

OtherOthercomponentscomponents

For all except average,For all except average,“gray” = 0,“gray” = 0,

“white” > 0,“white” > 0,““black” < 0black” < 0 ،نوعی رنجبر نوعی، دکتر رنجبر دکتر


IMAGE COMPRESSION USING SVD The image is stored as a 256X264

matrix M with entries between 0 and 1 The matrix M has rank 256

Select r X 256 as an approximation to the original M As r in increased from 1 all the way to

256 the reconstruction of M would improve i.e. approximation error would reduce

Advantage To send the matrix M, need to send

256X264 = 67584 numbers To send an r = 36 approximation of M,

need to send 36 + 36*256 + 36*264 = 18756 numbers 36 singular values 36 left vectors, each having 256 entries 36 right vectors, each having 264 entries

شنبه، دو

شنبه، دو

3/9/903/9/90

4444

Courtesy: Courtesy: http://www.uwlax.edu/faculty/will/svd/compression/index.html

USING PCA FOR RECOGNITION

Store each person as coefficients of projection onto first few principal components

Compute projections of target image, compare to database (“nearest neighbor classifier”)

شنبه، دو

شنبه، دو

3/9/903/9/90

4545

max

0iEigenfaceimage

i

iia

max

0iEigenfaceimage

i

iia



TOTAL LEAST SQUARES

8. One final least squares application Fitting a line: vertical vs. perpendicular

error

شنبه، دو

شنبه، دو

3/9/903/9/90

4646





TOTAL LEAST SQUARES

Distance from point to line:

where n is normal vector to line, a is a constant

Minimize:

شنبه، دو

شنبه، دو

3/9/903/9/90

4747

any

xd

i

i

i

an

y

xd

i

i

i

i i

i

ii an

y

xd

2

22

i i

i

ii an

y

xd

2

22

نوعی رنجبر نوعی دکتر رنجبر ،،دکتر


TOTAL LEAST SQUARES

First, let’s pretend we know n, solve for a

Then

شنبه، دو

شنبه، دو

3/9/903/9/90

4848

ny

x

ma

any

x

i i

i

i i

i

1

2

2

ny

x

ma

any

x

i i

i

i i

i

1

2

2

ny

xan

y

xd

my

i

mx

i

i

i

ii

i

ny

xan

y

xd

my

i

mx

i

i

i

ii

i



TOTAL LEAST SQUARES

So, let’s define

and minimize

شنبه، دو

شنبه، دو

3/9/903/9/90

4949

my

i

mx

i

i

i

i

i

y

x

y

x~

~

my

i

mx

i

i

i

i

i

y

x

y

x~

~

i i

in

y

x2

~

~

i i

in

y

x2

~

~



TOTAL LEAST SQUARES

Write as linear system

Have An=0Problem: lots of n are solutions, including n=0Standard least squares will, in fact, return n=0

شنبه، دو

شنبه، دو

3/9/903/9/90

5050

0~~

~~

~~

33

22

11

y

x

n

n

yx

yx

yx

0~~

~~

~~

33

22

11

y

x

n

n

yx

yx

yx



CONSTRAINED OPTIMIZATION

9. Solution: constrain n to be unit length So, try to minimize |An|2 subject to |n|2=1

Expand in eigenvectors ei of ATA:

where the i are eigenvalues of ATA

شنبه، دو

شنبه، دو

3/9/903/9/90

5151

nnnnn

AAAAA TTT2 nnnnn

AAAAA TTT2

22

21

2

222

211

TT

2211

n

nn

n

AA

ee

22

21

2

222

211

TT

2211

n

nn

n

AA

ee



CONSTRAINED OPTIMIZATION

To minimize subject toset min = 1, all other i = 0

That is, n is eigenvector of ATA withthe smallest corresponding eigenvalue

شنبه، دو

شنبه، دو

3/9/903/9/90

5252

222

211 2

22211

122

21 12

221



APPLICATIONS OF SVD IN LINEAR ALGEBRA Homogeneous equations, Ax

= 0Minimum-norm solution is

x=0 (trivial solution)Impose a constraint ,“Constrained” optimization

problemSpecial Case

If rank(A)=n-1 (m ¸ n-1, n=0) then x= vn ( is a

constant)Genera Case

If rank(A)=n-k (m ¸ n-k, n-

k+1== n=0) then x=1vn-

k+1++kvn with 2

1++2n=1

شنبه، دو

شنبه، دو

3/9/903/9/90

5353

For proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79For proof: Johnson and Wichern, “Applied Multivariate Statistical Analysis”, pg 79

Axx 1min

1x

Has appeared beforeHas appeared before Homogeneous solution of Homogeneous solution of

a linear system of a linear system of equationsequations

Computation of Computation of Homogrpahy using DLTHomogrpahy using DLT

Estimation of Estimation of Fundamental matrixFundamental matrix

e igen & s ingular v alue d ecomposition قسمتی از درس ریاضی مهندسی...

Documents