chapter 9 function approximation 고려대학교 컴퓨터학과 음성정보처리 연구실...

Chapter 9Chapter 9Function ApproximationFunction Approximation

고려대학교 컴퓨터학과 음성정보처리 연구실고려대학교 컴퓨터학과 음성정보처리 연구실

2003010594 2003010594 조영규조영규 2004020594 2004020594 방규섭 방규섭 2005020594 2005020594 정재연정재연

KoreaUnivers

ity

2 음성정보처리연구실

Contents

9.1 Least Squares Approximation

9.2 Continuous Least Squares

9.3 Function Approximation at a Point

9.4 Using Matlab’s Functions

KoreaUnivers

ity


9.1 Least Squares Approximation

Some of the most common methods of approximating data are based on the desire to minimize some measure of the difference between the approximating function and the given data points.

The method of least squares seeks to minimize the sum of the squares of the differences between the function value and the data value.

Advantages to using the square of the differences at each point

Positive differences do not cancel negative differences

Differentiation is not difficult

Small differences become smaller and large differences are magnified

KoreaUnivers

ity


9.1.1 Linear Least Squares Approximation - Example 9.1 Linear Approximation to Four Points

( ) 0.9 1.4f x x (1,2.1),(2,2.9),(5,6.1),(7,8.3)

The total sqared error is 0.04 0.09 0.04 0.36 0.53

24 4 4At 7 : (7) 7.7, 8.3, (7.7 8.3) 0.36x f y e

23 3 3At 5 : (5) 5.9, 6.1, (5.9 6.1) 0.04x f y e

22 2 2At 2 : (2) 3.2, 2.9, (3.2 2.9) 0.09x f y e

21 1 1At 1: (1) 2.3, 2.1, (2.3 2.1) 0.04x f y e

KoreaUnivers

ity


9.1.1 Linear Least Squares Approximation - Discussion (1/2)

Normal equation for linear least square approximation (1/2)

( )f x ax b 1 1 2 2 3 3 4 4( , ),( , ),( , ),( , )x y x y x y x y2 2 2 2

1 1 2 2 3 3 4 4[ ( ) ] [ ( ) ] [ ( ) ] [ ( ) ]E f x y f x y f x y f x y

2 2 2 21 1 2 2 3 3 4 4[ ] [ ] [ ] [ ]ax b y ax b y ax b y ax b y

Simplifying gives2 2 2 2

1 2 3 4 1 2 3 4 1 1 2 2 3 3 4 4[ ] [ ]a x x x x b x x x x x y x y x y x y

1 2 3 4 1 2 3 4[ ] [1 1 1 1]a x x x x b y y y y

1 1 1 2 2 2 3 3 3 4 4 4[ ] [ ] [ ] [ ] 0E

ax b y x ax b y x ax b y x ax b y xa

1 1 2 2 3 3 4 4[ ] [ ] [ ] [ ] 0E

ax b y ax b y ax b y ax b yb

In general form2

1 1 1

n n n

i i i ii i i

a x b x x y

2

1 1 1 1

, , , n n n n

xx i x i xy i i y ii i i i

S x S x S x y S y

1 1

n n

i ii i

a x bn y

KoreaUnivers

ity


9.1.1 Linear Least Squares Approximation - Discussion (2/2)

Normal equation for linear least square approximation (2/2)

Matlab Function for linear Least Squares Approximation

The solution of the system of equations is

, xy x y xx y xy x

xx x x xx x x

nS S S S S S Sa b

nS S S nS S S

KoreaUnivers

ity


Example 9.2 Least Squares Straight Line to Fit Four Data Points

(1,2.1),(2,2.9),(5,6.1),(7,8.3) 1.0440, 0.9352a b

( ) 1.044 0.9352f x x

The total squared error is2 2 2 2

1 2 3 4 0.0360d d d d

Linear least squares straight line

KoreaUnivers

ity


Example 9.3 Noisy Straight-Line Data

[0.00 0.20 0.80 1.00 1.20 1.90 2.00 2.10 2.95 3.00]x

[0.01 0.22 0.76 1.03 1.18 1.94 2.01 2.08 2.90 2.95]y

0.9839 , 0.0174a b

( ) 0.9839 0.0174f x x

Data and Linear fit for noisy straight line

KoreaUnivers

ity


9.1.2 Quadratic Least Squares Approximation - Discussion (1/2)

Normal equation for Quadratic least square approximation

1 1 nFor n data points, ( , ),...,( , ) we wish to minimizenx y x y2 2 2 2

1 1 1[ ( ) ( ) ] ... [ ( ) ( ) ]n n nE a x b x c y a x b x c y

2 2 2 21 1 1 1[ ( ) ( ) ]( ) ... [ ( ) ( ) ]( ) 0n n n n

Ea x b x c y x a x b x c y x

a

2 21 1 1 1[ ( ) ( ) ]( ) ... [ ( ) ( ) ]( ) 0n n n n

Ea x b x c y x a x b x c y x

b

2 21 1 1[ ( ) ( ) ] ... [ ( ) ( ) ] 0n n n

Ea x b x c y a x b x c y

c

Simplifying gives

4 3 2 2

1 1 1 1

n n n n

i i i i ii i i i

a x b x b x x y

3 2

1 1 1 1

n n n n

i i i i ii i i i

a x b x c x x y

2

1 1 1

[ ]n n n

i i ii i i

a x b x c n y

KoreaUnivers

ity


9.1.2 Quadratic Least Squares Approximation - Discussion (2/2)

Matlab Function for Quadratic Least Squares Approximation

KoreaUnivers

ity


Example 9.6 Oil Reservoir

1668.9 360.3 82.8

360.3 82.8 21.3

82.8 21.3 7.0

A

130.3413

42.4743

21.0000

b

2( ) 0.299 2.9926 8.5687p x x x

Using the Matlab function for quadratic least squares, we find that the normal equations are Az=b, with

The solution is [0.2990, 2.9926,8.5687]z

KoreaUnivers

ity


9.1.3 Cubic Least Squares Approximation

6 5 4 3 3

1 1 1 1 1

5 4 3 2 2

1 1 1 1 1

4 3 2

1 1 1 1 1

3 2

1 1 1 1 1

,

,

,

1 .

n n n n n

i i i i i ii i i i i

n n n n n


n n n n n


n n n n n

i i i ii i i i i

a x b x c x d x x y

a x b x c x d x x y

a x b x c x d x x y

a x b x c x d y

3 2( )f x ax bx cx d

KoreaUnivers

ity


Example 9.7 Cubic Least Squares2.6259 0.0000 3.1328 0.0

0.0000 3.1328 0.0000 4.4

3.1328 0.0000 4.4000 0.0

0.0000 4.4000 0.0000 11.0

A

1.5226

2.2000

2.2800

5.5000

r

3 2( )p x ax bx cx d

0.2550, 0.0000, 0.6997, 0.5a b c d

KoreaUnivers

ity


Example 9.8 Cubic Least Squares, Continued

1.3130 1.4160 1.5664 1.8

1.4160 1.5664 1.8000 2.2

1.5664 1.8000 2.2000 3.0

1.8000 2.2000 3.0000 6.0

A

1.6613

1.9976

2.6400

4.6500

r

3 2( )p x ax bx cx d

0.00, 0.375, 0.825, 0.500a b c d

KoreaUnivers

ity


9.1.4 Least Squares Approximation for Other Functional Forms

If the data are best fit by an exponential function, it is convenient instead to fit the logarithm of the data by a straight line

KoreaUnivers

ity


Example 9.10 Least-Squares Approximation of a Reciprocal Relation

The plot of the following data suggests that they could be fit by a function of the form y=1/(ax+b)

x=[0 0.5 1 1.5 2]

y=[1.00 0.50 0.30 0.20 0.20]

KoreaUnivers

ity


To approximate the exact value of the function for all point in an interval

The summations are replaced by the corresponding integrals

To approximate a given function a quadratic function on the interval [0, 1] and [-1, 1],

we minimize the following equation.

9.2 Continuous Least Squares

( )s x

2( )p x ax bx c

1 2 2

0

1 2

0

1

0

1

0

[ ( )]

( )5 4 3

( )4 3 2

( )3 2

E ax bx c s x dx

a b cx s x dx

a b cxs x dx

a bc s x dx

1 2 2

1

1 2

1

1

1

1

1

[ ( )]

2 20 ( )

5 32

0 0 ( )3

20 2 ( )

3

E ax bx c s x dx

a c x s x dx

b xs x dx

a c s x dx

KoreaUnivers

ity


To find the continuous least squares quadratic approximation to the exponential function on [-1, 1]

The coefficient matrix

Example 9. 12 Continuous Least Squares

2 20

5 32

0 03

20 2

3

A

2 2

2 2

2

2

2( ) ( 2 2)

x x x

x x x x

x x x

x x

x e dx x e xe dx

x e xe e e x x

xe dx x e e

e dx e

1 2

1

1

1

1

1

50.8789

20.7358

12.3504

x

x

x

x e dx ee

xe dxe

e dx ee

KoreaUnivers

ity


Example 9. 12 Continuous Least Squares (cont.)

2 20

5 3 0.87892

0 0 0.73583

2.35042

0 23

z

0.5368

1.1037

0.9963

z

Exponential function2( ) 0.5368 1.1037 0.9963f x x x

2( ) 0.5 1t x x x

KoreaUnivers

ity


9.2.1 Continuous Least Squares with Orthogonal Polynomails

The set of functions {f0, f1, f2, ….., fn} is linearly independent on

the interval [a, b]A linear combination of the functions is the zero function only if all of the coefficients are zero

In other words c0f0(x) + c1f1(x) + c2f2(x) + …. + cnfn(x) = 0 for all x in [a, b],

then c0=c1=c2=….=cn=0.

Example of linearly independent function

f0=1, f1=x, f2=x2,…., fn=xn

{p0, p1, p2, ….., pn} where pj is a polynomial of degree j

{1, sin(x), cos(x), sin(2x), cos(2x),…., sin(nx), cos(nx)}

The set of functions {f0, f1, f2, ….., fn} is orthogonal on [a, b]

Specially, if the functions are orthogonal and d j =1 for all j, the functions

are called orthonormal.

0( ) ( )

0

b

i jaj

if i jf x f x dx

d if i j

KoreaUnivers

ity


9.2.2 Gram-Schmidt process

How to construct a sequence of polynomials that are arthogonal on the interval [a, b]

Conditions

Pn be a polynomial of degree n

The coefficient of xn in pn be positive

For n = 0, 1, 2, ….

( ) ( ) 0b

n map x p x dx if n m

( ) ( ) 1b

n map x p x dx

Starting by taking p0(x)=x>0

To satisfy condition 3

2 21 ( ) 1

1

b

ac dx b a c

cb a

To construct p1(x), we begin by letting q1(x) = x + c1,0p0.

q1(x) is orthogonal to p0

0 1,0 0

0 1,0 0 0

( ) 0

0

b

a

b b

a a

p x c p dx

xp dx c p p dx

KoreaUnivers

ity



To satisfy condition 3, we normalize q1(x)

To construct p2(x), we begin by letting q2(x) = x2 + c2,1p1(x) + x2,0p0(x). q2(x) is orthogonal to p1(x)

Normalizing q2(x)

1,0 0 0 0sin 1b b

a ac xp dx ce p p dx

11

1 1

( )( )

( ) ( )b

a

q xp x

q x q x dx

21 2,1 1 2,0 0

21 2,1 1 1 2,0 1 0

( )( ( ) ( )) 0

( ) ( ) ( ) ( ) 0

b

a

b b b

a a a

p x x c p x c p x dx

x p dx c p x p x dx c p x p x dx

22,1 1 1 1 1 0sin ( ) ( ) 1 ( ) ( ) 0

b b b

a a ac x p dx ce p x p x dx p x p x dx

22

2 2

( )( )

( ) ( )b

a

q xp x

q x q x dx

KoreaUnivers

ity



The process continues in the same manner, as we construct each higher degree polynomial in turn.

Where, qn(x) = xn + cn,0p0(x) + cn,1p1(x) + ….. + cn,n-1pn-1(x)

(The coefficients cn,0, … , cn,n-1 are found so that qn(x) is orthogonal to each of the previously generated polynomials)

( )( )

( ) ( )

nn b

n na

q xp x

q x q x dx

KoreaUnivers

ity


9.2.3 Legendre Polynomials

A function of x defined on any finite interval a≤ x≤ b To transformed to a function of t defined -1≤ t≤ 1

The continuous least squares function approximation of Legendre polynomials

Legendre polynomials form an orthogonal set on [-1, 1] For normalization

An alternative definition of the Legendre polynomials

2 2

b a b ax t

0 1

2 32 3

4 2 5 34 5

( ) 1, ( ) ,

1 3( ) , ( ) ,

3 56 3 10 5

( ) , ( )7 35 9 21

p x p x x

p x x p x x x

p x x x p x x x

1

0 01

1 1 21 11 1

1 1 4 22 21 1

( ) ( ) 2

2( ) ( )

32 1 8

( ) ( )3 9 45

p x p x dx

p x p x dx x dx

p x p x dx x x dx

20

( 1)( ) 1 ( ) [(1 ) ], 1

2 !

n nn

n n n

dp x and p x x for n

n dx

1

1

2( ) ( )

2 1n np x p x dxn

KoreaUnivers

ity


9.2.4 Least Squares Approximation with Legendre Polynomials

To find the quadratic least squares approximation to f(x) on t

he interval [-1, 1] , we need to determine the coefficient c0, c

1, c2 that minimize

Setting

1 20 0 1 1 2 21

[ ( ) ( ) ( ) ( )]E c P x c P x c P x f x dx

0

0E

c

1

0 0 1 1 2 2 012[ ( ) ( ) ( ) ( )] ( ) 0c P x c P x c P x f x P x dx

1 1 1 1

0 0 0 1 1 0 2 2 0 01 1 1 1( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )c P x P x dx c P x P x dx c P x P x dx f x P x dx

1 1

0 0 0 01 1( ) ( ) ( ) ( )c P x P x dx f x P x dx

KoreaUnivers

ity


9.2.4 Least Squares Approximation with Legendre Polynomials (cont.)

In a similar manner, equations formed by setting and

and

The integrations on the left were performed earlier; the result is

1

0E

c

2

0E

c

1 1

1 1 1 11 1( ) ( ) ( ) ( )c P x P x dx f x P x dx

1 1

2 2 2 21 1( ) ( ) ( ) ( )c P x P x dx f x P x dx

1

0 01

1

1 11

1

2 21

1( ) ( )

23

( ) ( )245

( ) ( )8

c f x P x dx

c f x P x dx

c f x P x dx

KoreaUnivers

ity


Example 9.13 Least Squares Approximation Using Legendre Polynomials

To find the quadratic least squares approximation to f(x) = ex

on the interval [-1, 1]in terms of the Legendre polynomials

g(x) = c0p0(x) + c1p1(x) + c2p2(x) where p0(x) = 1, p0(x) = x, p0(x) = x2 –1

3

1 1

01 1

1 1

11 1

1 1 221 1

1 1 1( ) (2.3504) 1.1752

2 2 23 3 3

( ) ( ) (0.7358) 1.10372 2 245 45 1 45 1

( ) ( ) ( ) (0.8789 2.3504) 0.53688 8 3 8 3

x x

x

x

e p x dx e dx

f x p x dx xe dx

f x p x dx e x dx

2

0 0 1 1 2 2 0 1 2

1( ) ( ) ( ) ( ) ( )

3g x c p x c p x c p x c c x c x

KoreaUnivers

ity


Example 9.13 Least Squares Approximation Using Legendre Polynomials

Exponential function2

0 1 2

1( ) ( )

3g x c c x c x

KoreaUnivers

ity


Example 9.14 Pade Approximation of the Runge function

Pade approximation of the runge functionThe Runge function

Its first three derivatives at x =0 are f(0)=1, f′(0)=0, f″(0)=-50, f‴(0)=0

Taylor polynomial of the function (at x=0)

To seed a rational-function representation, using k=3, m=1, and n=2

For k=3, the linear system of equation is

2

1( )

1 25f x

x

2 3( ) 1 0 25 0t x x x x

1 02

2 1

( )1

a x ar x

b x b x

KoreaUnivers

ity


9.4 Using Matlab’s Functions% script for polynomial regression

% generate data

X = -3 : 0.1 : 3;

y1 = sin(x);

y2 = cos(2*x);

yy = y1 + y2;

y = 0.01*round(100*yy)

% find least squares 6th-degree polynomial to fit data

z = polyfit (x, y, 6)

% evaluate t he polynomial

p = polyval(z, x);

% plot the polynomialplot(x, p)Hold on% plot the dataplot (x, y, ‘+’)hold off

chapter 9 function approximation 고려대학교 컴퓨터학과 음성정보처리 연구실...

Documents