adavanced numerical computation 2008, am ndhu1 function approximation linear models line fitting...

Adavanced Numerical Computation 2008, AM NDHU

1

•Function approximation•Linear models•Line fitting•Hyper-plane fitting

•Discriminate analysis

Least Square Method

Advanced Numerical Computation 2008, AM NDHU

2

Unconstrained optimization

A target function, y = g(x), where x Rd

A sample from the surface of f paired data S={(xi ,yi)}i

yi =g(xi )+i

Let G(x; ) be an approximating function to g collects built-in parameters Minimizing the mean square approximating

error induces an unconstrained optimization


3

Data driven function approximation


4



5

Mean square approximating error

)(minarg

));(()( 2

Sopt

iiiS

E

xGyE


6

Nonlinear System

i k

ii

i k

ii

kk

Tn

i

ii

i

ii

i

iii

iiiS

d

xGdxG

d

xdGy

d

dEf

ffF

d

xGdxG

d

xdGy

d

xdGxGy

d

dE

xGyE

);();(

);()(

0)](),...,([)(

);();(

);(

);());((

));((2

1)(

1

2


7

Minima

dE/d = 0Local minima : unsatisfied approximationGlobal minima : reliable and effective

approximation


8

Nonlinear system

The severe local minimum problem needs to be overcome

0)(

)(

d

dEF S


9

Approximating function

DimensionalityOne dimensional functionsHigh dimensional functionsExtremely high dimensional functions

Linear functions, quadratic functions and nonlinear functions


10

Line fitting

Given paired data, (xi, yi ), minimize

n

iii yaxaaa

1

20101 )(),E(

one dimensional linear function


11

Paired data

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 11.8

2

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6

n=100; x=rand(1, n);y=1.5*x+2+rand(1, n)*0.1-0.05;plot(x,y,'.')

2

5.1

0

1

a

a


12

Fitting criteria

n1,...,ifor ,01 axay ii

n1,...,ifor ,01 ii yaax


13

Pseudo Inverse

n1,...,ifor ,01 ii yaax

nn y

y

y

a

a

x

x

x

2

1

0

12

1

1

1

1

1


14

Form X and b

X=[x' ones(n,1)];b=y';


15

Line fitting

ba TT X)XX( 1

>> a=inv(X'*X)*X'*b

a =

1.4816 2.0113


16

Strategy I: Line fitting

ba )X(pinv

>> a=pinv(X)*b

a =

1.4816 2.0113

Matlab built-in function


17

Demo_line_fitting

demo_line_fitting.m


18

Stand alone executable file

mcc -m demo_line_fitting.m

demo_line_fitting.exedemo_line_fitting.ctf


19

Strategy II

Conjugate gradient method

bTT XXX


20

Comparison

pinv versus conjugate gradient methodd=1,n=100

d=1;n=100; x=rand(d, n); y=rand(1,d)*x+2+rand(1, n)*0.1-0.05; plot(x,y,'.'); X=[x' ones(n,1)]; b=y'; tstart = tic; a=pinv(X)*b telapsed = toc(tstart) tstart = tic; x0=rand(1,d+1)'; a2 = conjugate(X'*X,X'*b,x0) telapsed = toc(tstart)


21

Hyper-plane fitting

bXa

nmnnn

m

m

m

mn

xxxx

xxxx

xxxx

xxxx

321

3333231

2232221

1131211

X

ma

a

a

a

3

2

1

a

nb

b

b

b

3

2

1

b


22

Sampling

A sample from a hyper-plane or a cloud

A mapping from R2 to RPaired data:

iiiii sss )},,({ 321s

iiiii Ryxx }),,(|)y,{( 21ii xx

plane-hyper a from sampling as

mappinglinear of data pairedConsider

),,(

),,(

321

21

iii

iiii

sss

yxx

s


23

n=100, d=2General coordinate of points (s1,s2,s3) or

(x1,x2,y)Linea relation

s1a1+s2a2+a3=s3

equivalently

x1a1+x2a2+a3=y


24

d < n

The number of unknowns is less than the constraint numberOne hundred data points in R3 space

Data point on a hyper-planesi1a1+si2a2 +a3 - si3 =0

xi1a1+xi2a2 +a3 - yi =

Minimization of the mean square error ke

2e


baX

n1,...,ifor bi axTi

nmnnn

m

m

m

mn

xxxx

xxxx

xxxx

xxxx

321

3333231

2232221

1131211

X

ma

a

a

a

3

2

1

a

nn y

y

y

y

b

b

b

b

3

2

1

3

2

1

b

m=d+1 and 1imx


26

Strategy I : Pseudo Inverse

baX

ba X)(pinv


27

Strategy II: minimizing mean square errors

n

ii

Ti

n

ii b

ne

n

eE

Minimize

1

2

1

2

2

)(11

)(

ax

a


28

Minimization

m1,...,jfor 0)(

jda

dE a


29

Derivative

0)(2)(

1

n

iiji

Ti

j

xbnda

dEax

a

n

ii

Ti b

nE

Minimize

1

2)(1

)( axa


30

Vector Form

0)(2)(

1

m

iii

Ti b

nd

dExax

a

a

n

ii

Ti b

nE

Minimize

1

2)(1

)( axa


31

Linear system: normal equations

n

iii

n

ii

Ti b

11

xaxx

0)(2)(

1

n

iii

Ti b

nd

dExax

a

a


32

n

iii

n

i

Tii

n

iii

n

ii

Ti

b

b

11

11

xaxx

xaxx


33

Tn

T

T

x

x

x

2

1

X nT xxx 21X


34

Tj

jj

Tn

T

T

nT

xx

x

x

x

xxx

2

1

21XX


35

n

iii

n

i

Tii b

11

xaxx

bXXaXT T


36

bXXXa TT 1)(

bXXaXT T


37

Comparison

pinv versus conjugate gradient methodd=2,n=100

d=2;n=100; x=rand(d, n); y=rand(1,d)*x+2+rand(1, n)*0.1-0.05; plot(x,y,'.'); X=[x' ones(n,1)]; b=y'; tstart = tic; a=pinv(X)*b telapsed = toc(tstart) tstart = tic; x0=rand(1,d+1)'; a2 = conjugate(X'*X,X'*b,x0) telapsed = toc(tstart)


38

Hyper-plane fitting

Step 1. Input paired data, (xi , yi), i=1…n Step 2. Form matrix X and vector b Step 3. Set a to pinv(X)*b Step 4. Set c to bTT XXX 1)(

Hyper-plane fitting ax y


39

>> n=30;S=rand(n,2);y=S*[1 2]'+1;>> b=y;>> X=[S ones(n,1)];>> a=pinv(X)*b; c=inv(X'*X)*(X'*b);>> sum(abs(a-c))

ans =

1.0547e-015


40

Generate training dataGet S and y

Performance evaluation

Hyper-plane fitting

S,ya

Generate testing dataGet S_test and y_test

Generate y_hat

S_test

y_test

-

Error rate for testing

TRAINING TESTING


41

Generate y_hat

>> n=10;S_test=rand(n,2);y_test=S_test*[1 2]'+1;>> X_test = [S_test ones(n,1)];>> y_hat = X_test * a;


42

Testing error

>> error_rate= mean((y_test-y_hat).^2)


43

demo_hp_fitting>> demo_hp_fittinga1:1a2:2a3:3

a =

0.9959 2.0035 3.0141


44

HP Tool


45

HP Tool

MLP_Tool.mMLP_Tool.fig


46

Mesh

fstr=input('input a 2D function: x1.^2+x2.^2+cos(x1) :','s');fx=inline(fstr);range=2*pi;x1=-range:0.1:range;x2=x1;for i=1:length(x1) C(i,:)=fx(x1(i),x2);endmesh(x1,x2,C);


47

Nonlinear function approximation


48

Nonlinear function approximation

Target function & sample Unfaithful approximationby hyper-plane fitting


49

Linear projection

132121111 axaxay

132 211 xxy


50

Two linear projections

Add two linear projections

132)2,1( 211 xxyxxf

232)2,1( 212 xxyxxf

34)2,1( 121 xyyxxf


51

132)2,1( 211 xxyxxf 232)2,1( 212 xxyxxf

34)2,1( 1 xxxf


52

Post-nonlinear Projection

)tanh( 32211 axaxahy

tanh32211 axaxah y


53

)132tanh()2,1( 211 xxyxxf

)232tanh()2,1( 212 xxyxxf


54

Two post-nonlinear projections

)232tanh()132tanh()2,1( 2121 xxxxxxf


55



56



57

Linearly non-separable

• Classify blue and red dots to two categories• Linearly non-separable by hyper-plane fitting


58

Error rate

22.48 %


59

Classification

Discriminate analysisLinear discriminate analysis

win.rar178 paired data (s,y)s{R13} : predictor or features, the last 13 columny { 1,2,3} : three categories, the first column


60

Training & Testing data

win.dat

load win.datind=randperm(178)

win_train=win(ind(1:140),:);win_test=win(ind(141:178),:);

save win_train.mat win_train;save win_test.mat win_test;

win_train.matwin_test.mat


61

Load training dataGet S and y

Discriminate Analysis

Hyper-plane fitting

S,y

a

Load testing dataGet S_test and y_test

Generate y_hat

S_test

y_test

compare

Error rate for testing


62


63

Linear assumption

Predictor x=[x1,…,x13]T

y = a1*x1+a2*x2+…+a13*x13

Find a to

n

ii

Ti y

nE

Minimize

1

2)(1

)( axa


64

Demo_wine_fitting

Error Rate : 3.93%

adavanced numerical computation 2008, am ndhu1 function approximation linear models line fitting...

Documents