![Page 1: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/1.jpg)
Cojugate Gradient Method
Zhengru Zhang ( 张争茹 )
Office: Math. Building 413(West)
2010 年教学实践周 7.12-7.16
![Page 2: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/2.jpg)
Outline• Aim
• Method of Gauss Elimination
• Basic Iterative Methods
• Conjugate Gradient Method
– Derivation
– Theory
– Algorithm
• References
• Homework & Project
![Page 3: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/3.jpg)
AimSolve linear algebraic system like
a11 x1 + a12 x2 + ... + a1n xn = b1
a21 x1 + a22 x2 + ... + a2n xn = b2
...
an1 x1 + an2 x2 + ... + ann xn = bn
Using matrix, the above system can be written as
Ax=bA is a N x N matrix, b is a N x 1 vector
Consider the case: A is large and sparse
![Page 4: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/4.jpg)
Method of Gauss Elimination
![Page 5: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/5.jpg)
U = A, L = I
for k = 1 to N-1
for j = k +1 to N
ljk = ujk/ukk
uj,k:m = uj,k:m – ljkuk,k:m
Algorithm of Gaussian Elimination without Pivoting
• LU Factorization, let A=LU
• Solve Ly=b
• Solve Ux=y
![Page 6: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/6.jpg)
Operation Count of Gauss Elimination
• Gauss Elimination and Back Substitution
• There are 3 loops
• There are 2 flops per entry
• For each k, the inner loop is repeated for rows k +1, …, N
• Cost: about About N 3 flops23
23
![Page 7: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/7.jpg)
Instability of Gaussian Elimination without Pivoting
11
10
11
110 20
A1= A2=
Examples
Remedy• Pivoting• Partial Pivoting• Complete Pivoting
![Page 8: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/8.jpg)
Algorithm of Gaussian Elimination with Partial Pivoting
U = A, L = I
For k = 1 to N-1
for j = k +1 to N
ljk = ujk/ukk
uj,k:m = uj,k:m – ljkuk,k:m
![Page 9: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/9.jpg)
Basic Iterative Methods
• How to construct iterative sequence?
• Convergence? Conditions?
• Convergence rate?
![Page 10: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/10.jpg)
Jacobi iteration
X[k+1] = D-1(L+U) X[k] + D-1 bB = D-1(L+U)
Gauss Seidel iteration
X[k+1] = (D-L)-1 U X[k] + (D-L)-1 bB = (D-L)-1 U
• Iterative method X[k+1] = BX[k] +g converges if and only if
(B) < 1
Convergence rate
||X[k]-X*|| ||X[1]-X[0]||, where q =||B||<11
kq
q
![Page 11: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/11.jpg)
Steepest Decent Method• Consider the case: A is symmetric positive definite
• Quadratic functional
(x) = xTAx - 2bTx
• The solution of Ax=b is equivalent to find the minimizer of the functional (x)
• Method of optimization: find a direction pk and a step k
![Page 12: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/12.jpg)
Steepest Decent MethodDetermine pk and k
• Suppose that pk is determined. Let’s start from xk
• Let f() = (xk + pk)
= (xk + pk)TA(xk + pk)-2bT(xk + pk)
= 2pkTApk - 2 rk
Tpk + (xk)
where rk = b - Axk (Residual)
• By calculas f’() = 2pkTApk- 2rk
Tpk =0
• Then let xk+1 = xk + k pk
Tk k
k Tk k
r p
p Ap
![Page 13: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/13.jpg)
Algorithm for Steepest Decent Method
• Verify (xk+1) - (xk) = (xk + k pk) - (xk)
= k2pk
TApk - 2k rkTpk
• How to determine the direction pk ?
take as the negative gradient pk = rk
2( )0 (if 0)
TTk kk kT
k k
r pp Ap
p Ap
![Page 14: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/14.jpg)
Algorithm Convergence Theorem
Suppose the eigenvalues of A
then there holds
where
![Page 15: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/15.jpg)
Conjugate Gradient MethodDerivation
• Negative gradient direction rk is the locally
steepest
decent direction, but it may not be the global one
• Consider a new direction: combination of rk and
pk-1
• Initially, take p0 = r0 , x1 = x0 + 0p0
• For step k +1, choose and to minimize
• By calculas
![Page 16: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/16.jpg)
• The corresponding minimizer is
0 and 0 satisfy
take
Let
• In summary,
01
0k
0
![Page 17: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/17.jpg)
Algorithm for CG method
Operations involved:
• Transpose,
• Scalar Multiply,
• Matrix Add,
• Matrix Multiply
Where and are obtainedin a simple form
![Page 18: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/18.jpg)
Properties for CG method
Orthogonalproperties
• Theoretically, CG method is an exact method. Actually,
works as an iterative method.
• Convergence rate:
where
![Page 19: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/19.jpg)
References
• 徐树方,高立,张平文, 数值线性代数,北京大学出版社,北京, 2007
• 袁亚湘,孙文瑜, 最优化理论与方法,科学出版社,北京, 2000
• Yousef Saad, Iterative Methods for Sparse Linear Systems, 2000
![Page 20: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/20.jpg)
Home & ProjectDue at the end of this week
• Solve the following linear systems using CG method2 2
1
0
1(1 ) ( ) 1, 2, , 1
2 2 20
j j j j
n
h hu u u f j n
u u
where 2 2 1(1 4 )sin , j
jf h
n n
• Set n = 100, 200, 300, 400, 500
• Use Matlab to graph the solution (j, uj)
Problem: Minimize the functional E(u)=∫(|u|2+u2-2fu )dx
The corresponding Euler-Lagrange equation is
E/u=-2u+2u-2f=0 or -u+u=f
-u xx + u =f 0<x<1
f=(1+42)sin2x
u(0)=u(1)=0
![Page 21: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/21.jpg)
Home & ProjectDue at the end of this week
• Solve the following linear systems using CG method2 2
1, 1, , 1 , 1
,0 ,0
0, ,
1(1 ) ( )
4 4 4 , 1, 2, , 1
0, 0,1, 2 ,
0, 0,1,2 ,
ij i j i j i j i j ij
i n
j n j
h hu u u u u f
i j n
u u i n
u u j n
2 2 2 2 1
200( ( ) ) 200( ( ) ) 100(( ) )(( ) ), ij
j j i i i i i if h
n n n n n n n n n
• The unknowns can be ordered as below
,1 ,2 , 1 2,1 2,2 2, 1 1,1 1,2 1, 1, , , , , , , , ,i i i n n n n n nu u u u u u u u u
![Page 22: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/22.jpg)
• The coefficient matrix is of the block tridiagonal form
2 2( 1) ( 1)
n n
S B
B S B
B SA
B
B S
( 1) ( 1)
1
4 n nB I Where
S Tridiagonal matrix with
diagonal entry:
other entry:
2
14
h
1
4
• Set n=20,40,80,100. Find the solution
• Use Matlab to graph the solution (i, j, uij)
-u+u=f (x,y)(0,1)(0,1)
u(x,y)=100(x2-x)(y2-y)
f=200(y-y2) + 200(x-x2) + 100(x2-x)(y2-y)
![Page 23: Cojugate Gradient Method Zhengru Zhang ( 张争茹 ) zrzhang@bnu.edu.cn Office: Math. Building 413(West) 2010 年教学实践周 7.12-7.16](https://reader033.vdocuments.pub/reader033/viewer/2022061607/56649d1e5503460f949f1c80/html5/thumbnails/23.jpg)
The End