alg.pdf

7/27/2019 alg.pdf

1/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang

cCambridge Sci. Pub.

Numerical Algorithms

in Scientific Computing

Xin-She Yang

University of Cambridge, United Kingdom

Cambridge Scientific Publishers

7/27/2019 alg.pdf

2/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


c2009 Cambridge Scientific Publishers

All rights reserved. No part of this book may be reprinted orreproduced or utilised in any form or by any electronic, mechan-ical, or other means, now known or hereafter invented, includingphotocopying and recording, or in any information storage orretrieval system, without prior permission in writing form thepublisher.

British Library Cataloguing in Publication Data

A catalogue record for this book has been requested.

Library of Congress Cataloguing in Publication Data

A catalogue record has been requested

ISBN978-1-904868-66-8

Cambridge Scientific Publishers LtdP O Box 806Cottenham, Cambridge CB24 8QYUKwww.cambridgescientificpublishers.com

7/27/2019 alg.pdf

3/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


Chapter 2

Interpolation and

Numerical Integration

Interpolation forms an important part of numerical algorithmswhich are widely used in modern scientific computing and com-putational geometry. We will briefly review the standard inter-polation methods such as spline interpolation and Bezier curves.

2.1 Spline InterpolationThe basic concept of splines was formulated around the 1940s bySchoenberg, Bezier, de Boor, and de Casteljau. The spline inter-polation is used to construct a function, called spline function,of degree m for given n + 1 known values of yi at n + 1 datapoints xi(i = 0, 1, 2,...,n). These given points are organized inan increasing order so that

x0 < x1 < ... < xn. (2.1)

The values yi at the given data points are often called knot values.

The major requirement is that the constructed spline functionS(x) should be continuous and produce the exact value at thedata points. The spline function can be constructed in a piecewisemanner so that each spline function Si(x) is valid in each datainterval x [xi, xi+1].

In the simplest case of three points with given values (yi1,yi, yi+1) at three distinct points xi1, xi and xi+1, the simplest

15

7/27/2019 alg.pdf

4/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


16 Interpolation and Numerical Integration

s

s

s

(xi1, yi1)

(xi, yi)

(xi+1, yi+1)

linear

s

s

s

rrrj

derivatives

cubic

Figure 2.1: Splines for given values (yi1, yi, yi+1) andtheir derivatives (yi1, y

i, y

i+1) at three points.

splines are linear functions, which can be constructed in eachinterval [xi, xi+1] so that we have

Si(x) = yi +(yi+1 yi)(xi+1 xi)

(x xi), x [xi, xi+1], (2.2)

which corresponds to the piecewise line segments in Figure 2.1.

For two consecutive intervals, the functions Si1 and Si shouldbe continuous, that is to say, Si(xi) = Si1(xi) = yi. As i =0, 2,...,n 1, there are n such spline functions.

The linear spline interpolation is continuous, but not smoothas there is a discontinuity in the first derivatives. Now supposethe three derivatives yi1, y

i, y

i+1 are also given at these three

points (xi1, yi1), (xi, yi) and (xi+1, yi+1), can we construct aclass of better and smoother spline functions (such as the dashedcurve shown in Figure 2.1) so that their values and derivativesmeet the given conditions? The answer is yes, that is the cubicspline function. You may wonder why cubic?

In the interval [xi, xi+1], we now have four conditions: twofunction values yi, yi+1, and two derivatives y

i, y

i+1. Obviously,

a linear function is not enough, how about a quadratic functionS(x) = ax2 + bx + c? This function is relatively smooth and thefirst derivative could meet these requirements, but the secondderivative S(x) = a is constant. If we require that the second-derivatives are also continuous, this means that either the second

7/27/2019 alg.pdf

5/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


2.1 Spline Interpolation 17

derivatives are constant everywhere (thus a is the same every-where) or there is a discontinuity. Furthermore, we have onlythree unknowns a,b,c with four conditions, so in general theyare over-determined, and not all conditions will be met. Thus,we need a cubic function.

If we use the following generic cubic function

Si(x) = i(x xi)3 + i(x xi)2 + i(x xi) + i, (2.3)

where i = 1, 2,...,n, then the spline function is twice continuous

differentiable. Its first derivative is

Si(x) = 3i(x xi)2 + 2i(x xi) + i. (2.4)

The four conditions become

Si(xi) = yi, Si(xi+1) = yi+1, (2.5)

and

Si(xi) = yi, S

i(xi+1) = y

i+1. (2.6)

Thus, we have four equations and four unknowns i, i, i and i

which are uniquely determined. The general requirements amongdifferent intervals are Si(x), S

i(x), and S

i (x) should be continu-

ous. We have

Si(xi) = yi, Si(xi+1) = yi+1, i = 0, 2,...,n 1, (2.7)

Si(xi+1) = Si+1(xi+1) = y

(xi+1), i = 0, 1,...,n 2, (2.8)and

Si (xi+1) = Si+1(xi+1), i = 0, 1,...,n 2. (2.9)

For n intervals, we have 4n unknowns, but we have 4n 2 con-ditions: n + 1 from yi(i = 0, 1,...,n 1); 2(n 1) from y

i and S

i

(i = 0, 1,...,n 2); n 1 from Si , (i = 0, 1,...,n 2), so we need2 more conditions.

The two extra conditions are at the two end points i = 0 andi = n. The clamped boundary conditions are to set

S0(x0) = y0, S

n1(xn) = y

n, (2.10)

7/27/2019 alg.pdf

6/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang



which are usually given. If the derivatives at the end points arenot given, we can use the natural or free boundary conditions:

S0 (x0) = 0, Sn1(xn) = 0. (2.11)

In order to find the spline functions Si(x), it is conventionalto rewrite them in terms of the second derivatives i = S

i (x),

(i = 0, 1, 2, ..., n 1), and we have

i = 6i(x xi) + 2i. (2.12)

At x = xi, we havei(xi) = 2i, (2.13)

or

i =i2

. (2.14)

Using the continuity of i at x = xi+1, we have

i(xi+1) = i+1(xi+1) = 6ihi + 2i, hi = xi+1 xi. (2.15)

Combining with Eq.(2.14), we have

i =i+1

i

6hi . (2.16)

Since Si(xi) = yi at x = xi+1, we have

i = yi. (2.17)

Substituting i, i and i into Si(xi+1) = yi+1 at x = xi+1 andafter some rearrangement, we have

i =(yi+1 yi)

hi hi(i+1 4i)

6. (2.18)

Now substituting these coefficients into Eq.(2.2), we can expressSi in terms of the second derivatives i, and we have

Si(x) =1

6hi[i+1(x xi)3 + i(xi+1 x)3]

+[yi+1

hi hi

6i+1](x xi) + [ yi

hi hi

6i](xi+1 x). (2.19)

7/27/2019 alg.pdf

7/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


2.1 Spline Interpolation 19

This is a cubic polynomial, and the only thing left is to findthe coefficients i. Using the continuity conditions: Si(xi) =Si1(xi) = yi and S

i(xi) = S

i1(xi) at x = xi; Si(xi+1) =

Si+1(xi+1) = yi+1 and Si(xi+1) = S

i+1(xi+1) at x = xi+1, we

can rewrite the above equation as

hi1i1 + 2(hi1 + hi)i + hii+1 = 6[(yi+1 yi)

hi (yi yi1)

hi1],

where i = 1, 2,...,n 1. Writing them in a matrix form, we have

2(h0 + h1) h1 ... 0h1 2(h1 + h2) ... 0...

. . . hn20 ... hn2 2(hn2 + hn1)

12...

n1

=

6(y2y1h1

y1y0h0

)...

6(ynyn1hn1

yn1yn2hn2

)

. (2.20)

Since 0 = 0 and n = 0 are given from the natural boundary

conditions, this linear system will uniquely determine 1,...,n1.For any given set of data, we should solve the linear system toget i, to then compute Si(x). In the case of equal spacing h0 =hi = hn1 = h, the above equation becomes

4 1 ... 01 4 ... 0...

. . . 10 ... 1 4

12...

n1

=

6

h2

y2 2y1 + y0y3 2y2 + y1

...yn 2yn1 + yn2

. (2.21)

Example 2.1: For the function

y = f(x) = sin(x) + 2,

we now try to approximate it using cubic spline functions constructedfrom five points x0 = 0, x1 = /2, x2 = , x3 = 3/2, x4 = 2.We known that y0 = 2, y1 = 3, y2 = 2, y3 = 1, y4 = 2. Since

7/27/2019 alg.pdf

8/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang



0 1 2 3 4 5 60

1

2

3

c

c

c

c

c

y = sin(x)+ 2

Figure 2.2: Spline approximation to f(x) (dashed curve)using cubic splines (solid curves) and five data points.

these points are equally-spaced with h = /2, equation (2.21) thenbecomes

4 1 01 4 10 1 4

123

=

4.8634

0+4.8634

,

whose solution is

123

T

=

1.2159

0+1.2159

T

.

At the two end points, we use the natural boundary condition 0 = 0and 4 = 0. Now substituting 0, 1, 2, 3, 4 into Eq.(2.19) fori = 0, 1, 2, 3, we have

S0 = 0.1290x3 + 3.5012x + 2.0000, x [0,

2]

S1 = 0.1290( x)3

0.9549x + 5.000, x [

2 , ]

S2 = 0.1612(x )3 1.0343x + 5.2493, x [,3

2]

and

S3 = 0.1612(2 x)3 + 1.0343x 4.4987, x [3

2, 2].

7/27/2019 alg.pdf

9/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


2.2 Lagrange Polynomials 21

These spline functions are plotted in Figure 2.2 as the solid curvewhere the cubic spline curves almost fall on the exact curve (dashedcurve) of the original function y = sin(x) + 2.

2.2 Lagrange Polynomials

We have seen that the construction of spline functions is tedious.Lagrange polynomials provide a systematic way to construct in-terpolation functions.

For any given n points (xi, yi), (i = 1, 2,...,n), there is a La-grange interpolating polynomial P(x) of degree k (n1) whichpasses through all n points. That is

P(x) =n

i=1

Pi(x)yi, (2.22)

where

Pi =n

j=1,j=i

(x xj)(xi xj)

. (2.23)

For example, for n = 5, we have

P(x) = (x x2)(x x3)(x x4)(x x5)y1(x1 x2)(x1 x3)(x1 x4)(x1 x5)

+(x x1)(x x3)(x x4)(x x5)y2

(x2 x1)(x2 x3)(x2 x4)(x2 x5)

+(x x1)(x x2)(x x4)(x x5)y3

(x3 x1)(x3 x2)(x3 x4)(x3 x5)

+(x x1)(x x2)(x x3)(x x5)y4

(x4 x1)(x4 x2)(x4 x3)(x4 x5)

+(x x1)(x x2)(x x3)(x x4)y5

(x5

x1)(x5

x2)(x5

x3)(x5

x4). (2.24)

Example 2.6: For equally-spaced five points (xi, yi) = P1(0, 1),P2(1, 1), P3(2, 2), P4(3,0.5), and P5(4, 1), the Lagrange poly-nomial becomes

P(x) =(x 1)(x 2)(x 3)(x 4)

24y1

x(x 2)(x 3)(x 4)6

y2

7/27/2019 alg.pdf

10/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang



2 40

2

4

2

s

s

s

ss

Figure 2.3: Lagrange polynomials of degree k = 4 for 5given points, and degree k = 3 for consecutive four points.

+x(x 1)(x 3)(x 4)

4y3

x(x 1)(x 2)(x 4)6

y4

+x(x 1)(x 2)(x 3)y5

24,

which is equivalent to

P(x) = 12

x4 154

x3 + 476

x2 203

x + 1.

This polynomial and the five points are plotted in Figure 2.3. If weuse any consecutive four points, we can construct Lagrange polyno-mials of degree k = 3. We have

(x) = 34

x3 +11

4x2 2x + 1,

for P1, P2, P3 and P4; and

(x) =5

4x3

37

4x2 + 20x

11,

for P2, P3, P4 and P5. These two polynomials are also plotted inthe same figure. We can see that Lagrange polynomials can havestrong oscillations, depending on the degrees of polynomials.

The disadvantage of Lagrange polynomials is that when n in-creases, the order of the polynomials also increases, which leads

7/27/2019 alg.pdf

11/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


2.3 Bezier Curve 23

Exi1

Tf(x)s

s

s

quadratic

xi xi+1E

xi1

Tf(x)s

s

s

s

cubic

xi xi+1 xi+2

Figure 2.4: Bezier interpolation: quadratic and cubic.

to greater oscillations between data points. For equally-spacedpoints, the Lagrange interpolation oscillates around the true func-tion. However, the advantages of Lagrange polynomials are thatthey are unique and rigorous, and thus they become handy inmathematical proofs. In addition, they form the basic formula-

tion for shape functions in finite element analysis and they arealso widely used in signal processing including audio-video anal-ysis.

2.3 Bezier Curve

We now know that linear and quadratic spline functions are notquite smooth, but splines of higher degrees are not straightfor-ward to construct. There is an alternative way to constructsmooth interpolation functions, that is to use Bezier curves (see

Figure 2.4). Bezier interpolation was invented by P. Bezier in the1960s for shape design, which makes it possible to design mod-ern aerodynamically-efficient cars. These interpolation curves aresmooth and can easily be extended to higher dimensions to con-struct surfaces and volumes. Therefore, they are widely used inengineering, computer graphics, and sciences.

The quadratic Bezier curve for any three points: P0(xi1,

7/27/2019 alg.pdf

12/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang



r1

c2

c3s4r1

c2

c3

s4 r1

c2

c3

s4

Figure 2.5: Flexibility of Bezier curves.

yi1), P1(xi, yi), P2(xi+1, yi+1) as shown in Figure 2.4, can beconstructed using a parameter t [0, 1]

P(t) = (1 t)2P0 + 2t(1 t)P1 + t2P2, (2.25)which is equivalent to

x(t) = (1 t)2xi1 + 2t(1 t)xi + t2xi+1, (2.26)and

y(t) = (1

t)2yi1 + 2t(1

t)yi + t3yi+1. (2.27)

Clearly, t = 0 corresponds to (xi1, yi1) (end point), while t = 1gives (xi+1, yi+1) (another end point). The only unusual featureis that the curve does not go through the point (xi, yi). Thismight be a disadvantage as it is not an exact interpolation, how-ever, it becomes an advantage as the curve is very smooth andtangential to both end points. This characteristic is also true forBezier curves of higher degrees.

For four points, the cubic Bezier curve leads to

x(t) = (1 t)3xi1+ 3t(1t)2xi + 3t2(1t)xi+1+ t3xi+2, (2.28)

andy(t) = (1 t)3yi1 + 3t(1 t)2yi + 3t2(1 t)yi+1 + t3yi+2, (2.29)

where t [0, 1]. Cubic Bezier curves are very versatile in approx-imating various curves, and some examples are shown in Fig. 2.5.

It is straightforward to extend such curves to any degree nusing coefficients of the binomial expansion [(1 t) + t]n.

7/27/2019 alg.pdf

13/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


2.4 Numerical Integration 25

2.4 Numerical Integration

Differentiation is easier than integration in the sense that you canget the explicit expressions of derivatives of most functions andcomplicated expressions if they exist, while it is very difficult andsometimes impossible to express an integral in an explicit form,even for seemingly simple integrands. For example, the errorfunction, widely used in engineering and sciences, is defined as

erf(x) =2

x0

et2

dt. (2.30)

The integration of this simple integrand exp(t2) does not lead toany simple explicit expression, which is why it is often written aserf(), referred to as the error function. From advanced calculus,we know that

erf(0) = 0, erf() = 1. (2.31)For any other values, say erf(0.5) 0.52049, we have to usenumerical integration.

The beauty of a closed-form expression or a simple formula isthat it will give tremendous insight into the problem. However,

such closed-form formulas are rarely possible, especially for real-world processes. In most cases, only approximate solutions arepossible. Such approximations can be obtained by using simpli-fied models or solving the mathematical models using approxi-mation techniques. Numerical integration is essential to many ofthese approximations.

In some sense, the numerical integration can be thought ofas the reverse of an interpolation process. In an interpolation,we intend to construct a smooth curve (such as splines) fromthe nodal values given at particular points or locations. On theother hand, we know the exact form of the function (integrand)

in numerical integration, but our aim is to select the samplingpoints so that the integral (such as area) can be estimated usingthe nodal values at these sampling points. The most importantissue is probably how to choose sampling points and/or where todistribute these points in the given domain. The simplest way isto distribute the sampling points uniformly over a given interval,which leads to the trapezium rule and Simpsons rule.

7/27/2019 alg.pdf

14/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


Index

Bezierlinear, 23

quadratic, 231-D, 79

algorithms, 80annealing schedule, 178

ant algorithm, 155ant colony optimization, 156,

157

attractiveness, 172attractor, 14

bee algorithm, 152bioluminescence, 169

bisection method, 5Boltzmann constant, 126

Box-Muller method, 124

brightness, 171Buffons needle, 127

Cartesian distance, 173

cellular automata, 99, 104

Game of Life, 100

stochastic, 101central difference, 82

change of variables, 134chaos, 13

Chebyshev polynomial, 114

conjugate gradient method,67

constrained optimization, 149

convergence, 178

correlation coefficient, 109cost function, 149

Cramers rule, 55

decision variable, 149

design variable, 149

double bridge problem, 159

eigenvalue, 37, 73

eigenvector, 37

elliptic equation, 94

equality, 149error function, 25

Euler scheme, 80

FA, 170, 177

FA variant, 177

finite difference method, 79

finite element analysis, 53firefly algorithm, 152, 169,

170

fitness, 171

Gauss elimination, 55

Gauss-Jordan elimination, 58

Gauss-Seidel iteration, 95Gaussian integration, 29

Gerschgorins theorem, 41, 50

gradient-based method, 142

185

7/27/2019 alg.pdf

15/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


186 INDEX

harmonics, 164harmony search, 152, 163, 165Hessian matrix, 144heuristic, 155, 163histogram, 123hyperbolic equation

first-order, 87second-order, 88

implicit scheme, 90

importance sampling, 134inequality, 149inf, 140integral

multi-dimensional, 133interpolation, 15

Bezier, 15, 23

cubic spline, 16Lagrange polynomial, 21linear spline, 15

inverse transform method, 121

iteration, 143iteration method, 10, 67, 95

Gauss-Seidel, 73Jacobi, 69relaxation method, 74

Lagrange multiplier, 150Lagrange polynomial, 21

leap-frog scheme, 81light intensity, 172linear algebra, 53

linear generatorcongruential, 117Fibonacci, 118

linear programming, 54, 149linear system, 54low-discrepancy sequence, 137

LU decomposition, 59

LU factorization, 59

Marsaglia generator, 119

Mersenne twister, 120metaheuristic method, 155,

156

metaheuristics, 152, 163

method of least square, 106Metropolis algorithm, 125

modified Newtons method,

144modulus, 118Monte Carlo integration, 130

error, 136

Monte Carlo method, 127multi-peak function, 161, 177music-inspired, 163

nature-inspired, 152

Newtons method, 7

Newton-Raphson, 7, 11, 76

nonlinear equation, 75nonlinear iteration, 13

nonlinear optimization, 149normal distribution, 122NP-hard problem, 160numerical instability, 13

numerical integration, 25

Gauss quadrature, 30integration point, 30

Simpsons rule, 28trapezium rule, 27

objective, 143ODE, 83optimization, 10, 149

constrained, 148

hill-climbing, 144

Lagrange multiplier, 150

7/27/2019 alg.pdf

16/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


INDEX 187

linear, 149Newtons method, 142nonlinear, 149steepest descent, 144unconstrained, 141

orthogonal matrix, 61oscillation, 13

particle swarm optimization,163

PDE, 87, 104pheromone, 156, 159pitch adjusting, 165positive definite matrix, 47preconditioning, 68PRNG, 117probabilistic search, 155probability, 157pseudo code, 157, 171pseudo-random number, 117PSO, 175

QR decomposition, 61quadratic function, 143quasi-Monte Carlo, 136quasi-random number, 137

random number generator, 117random search, 175randomization, 165root-finding algorithm, 3Rosenbrocks function, 167Runge-Kutta method, 80, 82,

84

self-organized, 156, 170semi-definite matrix, 48series expansion, 172shooting method, 83simple iteration, 3

Simpsons rule, 27simulated annealing, 126Sobol sequence, 138spectral method, 96spectral radius, 49, 72spline, 15square matrix, 38stability condition, 81, 88, 90statistics, 105

linear regression, 106

sample mean, 105sample variance, 105

stochastic, 166stochastic search, 155stratified sampling, 135sup, 140

time-stepping, 90implicit, 81

trapezium rule, 27travelling salesman problem,

160

uniform distribution, 119, 123upwind scheme, 88

virtual ant algorithm, 161

wave equation, 88, 103Wolframs class, 101

Ziggurat algorithm, 121

7/27/2019 alg.pdf

17/17

N

Alg

orith

ms inScientifi

cC

omputin

g

(2009)

Xin-She Yang


alg.pdf

Documents