lec3 -

8/10/2019 lec3 -

1/38

8/10/2019 lec3 -

2/38

Outline

Solution Existence and Uniqueness

Gaussian Elimination Basics

LU factorization

Pivoting and Growth

Hard to solve problems

Conditioning

8/10/2019 lec3 -

3/38

SMA-HPC 2003 MIT

ApplicationProblems

[ ]

n sG V I

x bM

=

No voltage sources or rigid struts

Symmetric and Diagonally Dominant

Matrix is n xn

8/10/2019 lec3 -

4/38

SMA-HPC 2003 MIT

Systems of LinearEquations

1 1

2 2

1 2 N

N N

x b

x bM M M

x b

=

1 1 2 2 N Nx M x M x M b+ + + =

Find a set of weights, x, so that the weightedsum of the columns of the matrix M is equal to

the right hand side b

8/10/2019 lec3 -

5/38

SMA-HPC 2003 MIT


Key Questions

Given Mx = b

Is there a solution?

Is the solution Unique?

Is there a Solution?

1 21 2 NNM M bx x x+ + + =

1There exists weights, suc, h that,Nx x

A solution exists when b is in the

span of the columns of M

8/10/2019 lec3 -

6/38

SMA-HPC 2003 MIT


Key QuestionsContinued

Is the Solution Unique?

1 21 2 0NNM My y y M+ + + =

1Suppose there exists weigh , , not all zerots, Ny y

( )Then if , thereforex b M x y b= + =

A solution is unique only if the

columns of M are linearlyindependent.

8/10/2019 lec3 -

7/38

SMA-HPC 2003 MIT


Key Questions

Given Mx = b, where M is square If a solution exists for anyb, then the

solution for a specificb is unique.

Square Matrices

For a solution to exist for any b, the columns of M must

span all N-length vectors. Since there are only N

columns of the matrix M to span this space, these vectors

must be linearly independent.

A square matrix with linearly independentcolumns is said to be nonsingular.

8/10/2019 lec3 -

8/38

SMA-HPC 2003 MIT

Gaussian Elimination Method for Solving M x = b

A Direct MethodFinite Termination for exact result (ignoring roundoff)

Produces accurate results for a broad range of

matrices

Computationally Expensive

Important PropertiesGaussianElimination Basics

8/10/2019 lec3 -

9/38

8/10/2019 lec3 -

10/38

SMA-HPC 2003 MIT

Key Idea

11 1 12 2 13 3 1x M x M x b+ + =

21 1 22 2 23 3 2x M x M x b+ + =

31 1 32 2 33 3 3x M x M x b+ + =

Reminder by Example

Use Eqn 1 to Eliminate From Eqn 2 and 31x

31 1 32 2 33 3 3x M x M x b+ + =

3111 1 12 2 13 3 1

11

( )M

x M x M x bM

+ + =

2111 1 12 2 13 3 1

11

( )M

x M x M x bM

+ + =21 21 21

22 12 2 23 13 3 2 1

11 11 11

M M MM x M M x b b

M M M

+ =

31 31 3132 12 2 33 13 3 3 1

11 11 11

M M MM x M M x b b

M M M

+ =

GaussianElimination Basics

8/10/2019 lec3 -

11/38

SMA-HPC 2003 MIT

Key Idea in the Matrix

11

11 12 13

21 21 2122 12 23 12 2 2 1

11 11 11

31 31

32 12 33 12

311 113 3

0

0

xb

M M M

M M MM M M x b b

M M M

M M

M M M M MM M x b

=

1 111

bM

Reminder by Example

MULTIPLIERSPivot


8/10/2019 lec3 -

12/38

SMA-HPC 2003 MIT

Remove x2 from Eqn 3

Reminder by Example

Pivot

Multiplier


11 12 13

21 2122 12 23 12

11 11

3132 12

1131 2133 12 23 12

11 112122 12

11

0

0 0

M M M

M MM M M M

M M

MM M

MM MM M M M

M MMM M

M

11

212 1

11

2

3132 12

1131 213 1 2 1

11 112122 12

113

x b

Mb b

Mx

MM M

MM Mb b b b

M MMM M

Mx

=

8/10/2019 lec3 -

13/38

SMA-HPC 2003 MIT

11

11 12 13

21 21 2122 12 23 12 2 2 1

11 11 11

31 31

32 12 33 12311 11

3 3

0

0

xb

M M M

M M MM M M x b b

M M M

M M

M M M M MM M x b

=

1 111

bM

Simplify the Notation

Reminder by ExampleGaussian

Elimination Basics

22

23

32 33 3b

2b

8/10/2019 lec3 -

14/38

SMA-HPC 2003 MIT

Remove x2 from Eqn 3

Reminder by Example

Pivot

Multiplier


22 23 2

32 3233 23 3 2

111 12 13

2 2

1

2

3

2 2

0

0 0

x

M M b

M MM b b

M M

bM

x

M

x

M

=

8/10/2019 lec3 -

15/38

SMA-HPC 2003 MIT

11

2

3

22

11 12 1

2

3

2

3 3

3

30

0 0

x

M b

b

M

M M M

x

x b

=

Altered

During

GE

GE Yields Triangular System


Elimination Basics

33

33

yx

U=

2 23 32

22

y U xx

U

=

1 12 2 13 31

11

y U x U xx

U =

11 12 13 1 1

22 23 2 2

33 3 3

0

0 0

U U U x y

U U x y

U x y

=

8/10/2019 lec3 -

16/38

SMA-HPC 2003 MIT

The right-hand side updates


Elimination Basics

1

1

212 12

11

3 32 2

2

13 3 1

11 2

by

Mb by

M

M Mby b M Mb

=

1

1

212 12

11

3 322

2

13 3 1

11 2

by

Mb by

M

M Mby b

M Mb

=

32

1 1

212 2

11

3 3

31

11 22

1 0 0

1 0

1

y bM

y bM

y bMM

M M

=

8/10/2019 lec3 -

17/38

SMA-HPC 2003 MIT

3 2

3 2

2 1

1 1

3 1

1 1

1

1

1

M

M

M

M

M

M

Fitting the pieces together


Elimination Basics

22 2

11 12 13

3

3

3

M M M

M

M M

22 23

32

2

3

11 12 13

21

1

313

1

1 21

M

M M

M M

M

M M

M

8/10/2019 lec3 -

18/38

SMA-HPC 2003 MIT

Basics of LUFactorization

Solve M x = bStep 1

Step 2 Forward Elimination

Solve L y = b

Step 3 Backward Substitution

Solve U x = y

L U= i

=

Recall from basic linear algebra that a matrix A can be factored into the product of a

lower and an upper triangular matrix using Gaussian Elimination. The basic idea of

Gaussian elimination is to use equation one to eliminate x1 from all but the first

equation. Then equation two is used to eliminate x2 from all but the second

equation. This procedure continues, reducing the system to upper triangular form as

well as modifying the right- hand side. More is

needed here on the basics of Gaussian elimination.

8/10/2019 lec3 -

19/38

SMA-HPC 2003 MIT


Matrix

Solving TriangularSystems

111 1

221 22 2

13

1

0 0

0

NN NN N

l y b

l l y b

l

l l y b

=

The First Equation has onlyy1 as an unknown.The Second Equation has onlyy1 andy2 as

unknowns.

8/10/2019 lec3 -

20/38

SMA-HPC 2003 MIT


Algorithm

Solving Triangular

Systems

111 1

221 22 2

13

1

0 0

0

NN NN N

l y b

l l y b

l

l l y b

=

1

1 1

1

1 bl

y =

2

22

2

1( )

ly b=

121l y

3

33

3

1( )

ly b=

131l y 232l y

1

( )NN

N

Ny bl=

1

1

i

N

N i

il y

=

Solving a triangular system of equations is straightforward but expensive.y1 can be

computed with one divide,y2 can be computed with a multiply, a subtraction and a

divide. Onceyk-1 has been computed,ykcan be computed with k- 1multiplies, k- 2

adds, a subtraction and a divide. Roughly the number of arithmetic operations is

N divides + 0 add /subs + 1 add /subs + .N- 1 add /sub

fory1 fory2 foryN

+ 0 mults + 1 mult + .. + N- 1 mults

fory1 fory2 foryN

= (N- 1)(N- 2) / 2 add/subs + (N- 1)(N- 2) / 2 mults +Ndivides

= OrderN2 operations

8/10/2019 lec3 -

21/38

SMA-HPC 2003 MIT

11 12 13 14

21 22 23 24

31 32 33 34

41 42 43 44

M M M M

M M M M

M M M M

M M M M

444343 444 3

3 344

343332

Basics of LUFactorization Picture

Factoring

22 23 24

42

3 2

2 2

4 2

2 2

33 34

21

11

31

11

41

11

The above is an animation of LU factorization. In the first step, the first equation is

used to eliminate x1 from the 2nd through 4th equation. This involves multiplying

row 1 by a multiplier and then subtracting the scaled row 1 from each of the target

rows. Since such an operation would zero out the a21, a31 and a41 entries, we can

replace those zerod entries with the scaling factors, also called the multipliers. For

row 2, the scale factor is a21/a11 because if one multiplies row 1 by a21/a11 andthen subtracts the result from row 2, the resulting a21 entry would be zero. Entries

a22, a23 and a24 would also be modified during the subtraction and this is noted by

changing the color of these matrix entries to blue. As row 1 is used to zero a31 and

a41, a31 and a41 are replaced by multipliers. The remaining entries in rows 3 and 4

will be modified during this process, so they are recolored blue.

This factorization process continues with row 2. Multipliers are generated so that

row 2 can be used to eliminate x2 from rows 3 and 4, and these multipliers are

stored in the zerod locations. Note that as entries in rows 3 and 4 are modified

during this process, they are converted to gr een. The final step is to used row 3 to

eliminate x3 from row 4, modifying row 4s entry, which is denoted by convertinga44 to pink.

It is interesting to note that as the multipliers are standing in for zerod matrix

entries, they are not modified during the factorization.

8/10/2019 lec3 -

22/38

SMA-HPC 2003 MIT

LU BasicsAlgorithm

Factoring

For i = 1 to n-1 { For each Row

For j = i+1 to n { For each target Row below the source

For k = i+1 to n { For each Row element beyond Pivot

}}

}

Pivotji

ji

ii

MM

M=

Multiplier

jk jk ji ikM M M

8/10/2019 lec3 -

23/38

Swapping rowj with row i at step i ofLUfactorization is identical to applyingLU

to a matrix with its rows swapped a priori.

To see this consider swapping rows before beginningLU.

Swapping rows corresponds to reordering only the equations. Notice that

the vector of unknowns is NOT reordered.

SMA-HPC 2003 MIT

LU BasicsZero Pivots

Factoring

At Step i

Multipliers

Factored Portion

(L)

ii

ji

Row i

Row j

What if Cannot form0 ?iiM = ji

iiSimple Fix (Partial Pivoting)

IfFind

0iiM =0ji j i >

Swap Rowj with i

1 1 1

2 2 2

| |

| |

| |

i j

i j i

N N

r x b

r x b

r x b

r x b

r x b

=

8/10/2019 lec3 -

24/38

SMA-HPC 2003 MIT

LU BasicsZero Pivots

Factoring

Two Important Theorems

1) Partial pivoting (swapping rows) always succeeds

if M is non singular

2) LU factorization applied to a strictly diagonally

dominant matrix will never produce a zero pivot

Theorem Gaussian Elimination applied to strictly diagonally dominant matrices

will never produce a zero pivot.

Proof

1) Show the first step succeeds.

2) Show the (n- 1) x (n- 1) sub matrix

is still strictly diagonally dominant.

First Step as

Second row after first step

21 21 2122 12 23 13 2 1

11 11 11

0, , , , n na a a

a a a a a aa a a

21 2122 12 2 1

11 11

Is

?j ja a

a a a aa a

>

n

nn- 1

n- 1

11 0a 112

| | | |n

ij

j

a a=

>

8/10/2019 lec3 -

25/38

SMA-HPC 2003 MIT

LU BasicsSmall Pivots

Numerical Problems

Contrived Example

171

2

17

17 17

110 1

31 2

1 0 10 1L = U =

10 1 0 10 2

x

x

=

+

Can we represent

this ?

In order to compute the exact solution first forward eliminate

1

17

2

17

1 2

1 0 1

10 1 3

and therefore 1, 3- 10 .

y

y

y y

=

= =

171

17 172

17

2 17

Backward substitution yields

110 1

0 2 10 3 10

3 10and therefore 1

2 10

x

x

x

=

= +

17 17 1717 17

1 17 17

3 10 2 10 (3 10 )and 10 (1 ) 10 ( ) 1

2 10 2 10x

= = +

1 17

1 217

2

171

1 217 172

In the rounded case

1 0 11 10

10 1 3

110 11 0

0 10 10

yy y

y

xx x

x

= = =

= = =

8/10/2019 lec3 -

26/38

SMA-HPC 2003 MIT


Numerical Problems

64 bits

52 bits11 bitssign

Double precision number

Basic Problem

7

1.0 0.000000000000001 1.0

or

1.0000001 1 ( 10 ) r ight 8 digits

=

+

Key IssueAvoid small differences between large

numbers !

An Aside on Floating Point ArithmeticexponentX.X X X X X 10 i

8/10/2019 lec3 -

27/38

SMA-HPC 2003 MIT


Numerical Problems

Back to the contrived example

171

Exact 17 172

1 0 110 1LU

10 1 30 2 10

x

x

= =

171

Rounded 17 172

1 0 110 1LU

10 1 30 10

x

x

= =

1 1

2 2Exact Rounded

1 0

1 1

x x

x x

= =

8/10/2019 lec3 -

28/38

SMA-HPC 2003 MIT


Numerical Problems

Partial Pivoting for Roundoff reduction

If | | < max | |

Swap row with arg (max | |)

ii jij i

ijj i

M M

i M

>

>

reordered 17 17

1 0 1 2LU

10 1 0 1 2 10

= +

This multiplier

is small

This term gets

rounded

To see why pivoting helped notice that

The right hand side value 3 in the unpivoted case was rounded away, where as now

it is preserved. Continuing with the back substitution.

1

17

2

17

1 2

1 0 3

10 1 1

yields y 3, y 1 10 1

y

y

=

= = 17 17

2Notice that without partial pivoting was 3-10 or -10 with rounding.y

1

17

2

2 1

1 2 3

0 1 2 10 1

1 1

x

x

x x

= +

8/10/2019 lec3 -

29/38

SMA-HPC 2003 MIT


Numerical Problems

If the matrix is diagonally dominant or partial pivoting

for round-off reduction is during

LU Factorization:

1) The multipliers will always be smaller than one in

magnitude.

2) The maximum magnitude entry in the LU factors

will never be larger than 2(n-1)

times the maximummagnitude entry in the original matrix.

To see why pivoting helped notice that

The right hand side value 3 in the unpivoted case was rounded away, where as now

it is preserved. Continuing with the back substitution.

1

17

2

17

1 2

1 0 3

10 1 1

yields y 3, y 1 10 1

y

y

=

= = 17 17

2Notice that without partial pivoting was 3-10 or -10 with rounding.y

1

17

2

2 1

1 2 3

0 1 2 10 1

1 1

x

x

x x

= +

8/10/2019 lec3 -

30/38

SMA-HPC 2003 MIT

Hard to SolveSystems

Fitting Example

Polynomial Interpolation

Table of Data

t0 f(t0)

t1 f(t1)

tN f(tN)

f

tt0 t1 t2 tN

f(t0)

Problem fit data with anNth order polynomial2

0 1 2( ) N

Nf t t t t = + + + +

8/10/2019 lec3 -

31/38

SMA-HPC 2003 MIT


Example Problem

Matrix Form2

0 0 0 0 0

2

1 11 1 1

2

interp

1 ( )

( )1

( )1

N

N

NN N

N N N

t t t t

tt t t

tt t t

M

=

The kth row in the system of equations on the slide corresponds to insisting that the

Nth order polynomial match the data exactly at point tk. Notice that we selected the

order of the polynomial to match the number of data points so that a square system

is generated. This would not generally be the best approach to fitting data, as we

will see in the next slides.

8/10/2019 lec3 -

32/38

SMA-HPC 2003 MIT


Fitting Example

Coefficient

Value

Coefficient number

Fitting f(t) = t

f

t

Notice what happens when we try to fit a high order polynomial to a function that is

nearly t. Instead of getting only one coefficient to be one and the rest zero, instead

when 100th order polynomial is fit to the data, extremely large coefficients are

generated for the higher order terms. This is due to the extreme sensitivity of the

problem, as we shall see shortly.

8/10/2019 lec3 -

33/38

SMA-HPC 2003 MIT


Perturbation Analysis

Vector Norms

L2 (Euclidean) norm :

Measuring Error Norms

=

=n

i

ixx1

2

2

L1 norm :

L norm :

=

=n

i

ixx1

1

ii

xx max=

12

8/10/2019 lec3 -

34/38

8/10/2019 lec3 -

35/38

SMA-HPC 2003 MIT

1x M M x x +1 M x x +



SinceM x - b = 0

( ) ( )

UnperturbedModels LU ModelsSolution

RightHandSideRoundoff Perturbation

M M x x M x M x M x M x b + + = + + + =

Perturbation Equation

1( ) ( )x M x x x M M x x = + = +

Taking Norms1

M

Relative Error Relation1

""

ConditionNumber

xx

M Mx

+

As the algebra on the slide shows the relative changes in the solutionx is bounded

by anM- dependent factor times the relative changes in M. The factor

was historically referred to as the condition number ofM, but that definition has

been abandoned as then the condition number is norm- dependent. Instead thecondition number ofMis the ratio of singular values ofM.

Singular values are outside the scope of this course, consider consulting Trefethen

& Bau.

1|| || || ||M

max

min

( )cond ( )

( )M

=

8/10/2019 lec3 -

36/38

SMA-HPC 2003 MIT



Geometric Approach is clearer

1 2 1 1 2 2[ ], Solving is findingM M M x b x M x M b= = + =

2

2|| ||

M

M

1

1|| ||

2

2|| ||

M

M

1

1|| ||

When vectors are nearly aligned, difficult to determine

how much of versus how much of1

2

Case 16

1 0

0 10

Case 16

6

1 1 10

1 10 1

Columns orthogonal Columns nearly aligned

1

2

1

2

8/10/2019 lec3 -

37/38

11 11 1 11 11 11 1

1 1

0 0

0

00 0

N N

NN N NN NN N NN NN

d a a d a d a

d a a d a d a

=

SMA-HPC 2003 MIT


Geometric Analysis

Polynomial Interpolation

4 8 16 32

1010

1015

1020

~314~106

~1013

~1020log(cond(M))

n

The power series polynomials

are nearly linearly dependent

2

1 1

2

2 2

2

1

1

1 N N

t t

t t

t t

1

1

1

t

t 2

t

Question Does row- scaling reduce growth ?

Does row- scaling reduce condition number ?

Theorem If floating point arithmetic is used, then row scaling (D M x = D b) will

not reduce growth in a meaningful way.

1|| || || || condition number of MM M

If

M

and

'

then

' No roundoff reduction

LU

LU

x b

D M x Db

x x

=

=

=

8/10/2019 lec3 -

38/38

Summary

Solution Existence and Uniqueness

Gaussian Elimination Basics

LU factorization

Pivoting and Growth

Hard to solve problems

Conditioning

lec3 -

Documents