code: 101mat4–101mt4b today’s topics normed linear (vector) space symmetric matrix … · 2020....

Code: 101MAT4–101MT4B

Today’s topics◮ Linear (vector) space◮ Normed linear (vector) space◮ Matrix and vector norms◮ Symmetric matrix, positive definite matrix◮ Gaussian elimination, Cholesky factorization, condition

number

1 / 29

Vector (linear) space V : For all u, v ∈ V and all α, β ∈ R,the linear combination αu + βv belongs to V . Next,

x + y = y + x ∀x , y ∈ V , (1)

x + (y + z) = (x + y) + z ∀x , y , z ∈ V , (2)

∃! 0̃ ∈ V x + 0̃ = x , ∀x ∈ V , (3)

∀x ∈ V ∃! − x ∈ V x + (−x) = 0̃, (4)

∀α ∈ R ∀x ∈ V αx ∈ V , (5)

1x = x , (6)

∀α, β ∈ R α(βx) = (αβ)x , (7)

α(x + y) = αx + αy , (8)

(α+ β)x = αx + βx . (9)

Examples: Ordered n-tuples, matrices of the same size m × n,polynomials of the degree n, functions, etc.

Remark: Vector space over C: α, β ∈ C.

2 / 29

Normed vector space XA real vector space X is called a normed vector (linear) space,if a real number ‖x‖, called the norm, is assigned to eachx ∈ X in such a way that

‖x‖≥ 0 ∀x ∈ X , (10)

‖x + y‖≤ ‖x‖+ ‖y‖ ∀x , y ∈ X , (11)

‖αx‖= |α| ‖x‖ ∀x ∈ X , ∀α ∈ R, (12)

‖x‖= 0 ⇒ x = 0̃. (13)

The inequality (11) is equivalent to

‖u − v‖ ≤ ‖u − w‖+ ‖w − v‖ ∀u, v ,w ∈ X . (14)

The inequality (11) (and (14)) is called the triangle inequality.

Remark: If X is a space over the complex numbers, thedefinition remains unchanged except for α ∈ C.

3 / 29

Vector norms x = (x1, x2, . . . , xn) ∈ Rn, or ∈ C

n

‖x‖1 =n∑

i=1

|xi | (taxicab norm, Manhattan norm),

‖x‖2 =

(n∑

i=1

|xi |2)1/2

(Euclidean norm),

‖x‖p =

(n∑

i=1

|xi |p)1/p

(p-norm), p ≥ 1,

‖x‖∞ = maxi∈{1,2,...,n}

|xi | (max-norm).

4 / 29

We limit ourselves to real vectors and matrices.(Nevertheless, the statemets are valid for complex matrices,too.)

The norm of an m × n matrix A induced by the vector norms:

‖A‖YmXn = max{x∈Xn: x 6=0}

‖Ax‖Ym

‖x‖Xn

, (15)

where Xn ⊂ Rn and Ym ⊂ R

m are vector spaces.

If y = Ax , where x ∈ Xn and y ∈ Ym,then ‖y‖Ym ≤ ‖A‖YmXn ‖x‖Xn .

If the norm ‖ · ‖ξ , where ξ stands for 1, 2, p, or ∞, is used inboth spaces, then the induced norm is denoted by ‖A‖ξ.In special cases, the calculation of ‖A‖ξ is simpler than in (15).

5 / 29

In particular,

‖A‖1 = maxk∈{1,2,...,n}

m∑

i=1

|aik |, ‖A‖∞ = maxi∈{1,2,...,m}

n∑

k=1

|aik |,

‖A‖2 = (̺(ATA))1/2 (spectral norm).

If A is real and symmetric, then

‖A‖2 =(̺(ATA))1/2 = (̺(A2))1/2 = ̺(A).

Frobenius norm (is not induced!) ‖A‖F =

(m∑

i=1

n∑

k=1

|aik |2)1/2

.

It holds ̺(A) ≤ ‖A‖ for the Frobenius and all the inducednorms. Moreover, for the n × n identity matrix I, ‖I‖ = 1 and‖I‖F =

√n, where ‖ · ‖ is an arbitrary induced norm.

In the vector space with a finite dimension, all the norms areequivalent, that is, for instance,

∃c1, c2 > 0 ∀x ∈ Rn c1‖x‖1 ≤ ‖x‖2 ≤ c2‖x‖1.

6 / 29

Scalar (inner) product in a vector space V (over R)The map (·, ·) : V × V → R that maps pairs of the elements ofV into R is called scalar (inner) product on V if for all x , y ∈ Vand all α ∈ R the following statements are valid

(y , x) = (x , y), (16)

(x + z, y) = (x , y) + (z, y), (17)

(αx , y) = α(x , y), (x , αy) = α(x , y), (18)

(x , x) ≥ 0, (19)

(x , x) = 0 ⇔ x = 0. (20)

Moreover, ‖x‖2 =√

(x , x) defines a norm on V .

Remark: The inner product has to be defined in a partlydifferent way for the vector space over the complex numbers.

7 / 29

Inner product of real vectors

(x , y) =n∑

k=1

xkyk , (21)

where x = (x1, . . . , xn) ∈ Rn and y = (y1, . . . , yn) ∈ R

n.

The Euclidean norm on Rn is defined through ‖x‖2 =

√(x , x).

Remark: The inner product of functions(u, v) =

∫ ba u(x)v(x)dx , where u, v ∈ C([a,b]), also comply

with (16)-(20).

8 / 29

Schwarz (Cauchy) inequality

|(x , y)| ≤ ‖x‖2 ‖y‖2 ∀x , y ∈ V .

Proof: y = 0 ⇒ the statement holds. If y 6= 0, then

0 ≤∥∥∥∥∥x − (x , y)

‖y‖22

y

∥∥∥∥∥

2

2

=

(x − (x , y)

‖y‖22

y , x − (x , y)‖y‖2

2

y

)

= ‖x‖22 − 2

(x , y)2

‖y‖22

+(x , y)2

‖y‖22

= ‖x‖22 − (x , y)2

‖y‖22

.

Remark: The proof is general because it uses only theproperties (17)-(18), not a particular definition of the innerproduct! The inequality is valid for the inner product offunctions, too.

9 / 29

Positive definite matricesA symmetric matrix A = (aij) ∈ R

n×n is called positive definite,if

(Ax , x) > 0, i.e.,n∑

i=1

n∑

j=1

aijxixj > 0 (22)

for all 0 6= x ∈ Rn (if (Ax , x) ≥ 0 . . . positive semidefinite;

if > 0, < 0 for some vectors . . . indefinite)

Characterization: A matrix A is positive definite, i.e, A is s.p.d.

⇔ all the eigenvalues of A are positive

⇔ expression (22) is an inner product

⇔ the leading principal minors∗ of A are all positive(Sylvester’s criterion)

⇔ A has a unique Cholesky decomposition A = LLT

(see later), where the diagonal elements of L arepositive

∗ The k th leading principal minor of a matrix A is the determinant ofits upper-left k × k sub-matrix.

10 / 29

Examples:

A =

(9 66 5

), B =

(9 66 4

), C =

(9 66 3

),

xTAx = 9x21 + 12x1x2 + 5x2

2 = (3x1 + 2x2)2 + x2

2 , A is s.p.d.xTBx = 9x2

1 + 12x1x2 + 4x22 = (3x1 + 2x2)

2, B is positivesemidefinite, B is not s.p.d.xTCx = 9x2

1 +12x1x2 +3x22 = (3x1 +2x2)

2 − x22 , C is indefinite.

The leading principal minors of A are 9 an 9, of B are 9 and 0,of C are 9 and -9.

11 / 29

Relationship between a linear system of equations and theminimization of a quadratic function

A = (aij) ∈ Rn×n is s.p.d., b ∈ R

n is a column vector:

Let g(x) =12(Ax , x)− (b, x) and let the minimum of g is

attained at x̂ , then it follows from (22) that

x̂ = arg minx∈Rn

g(x) ⇐⇒ grad g(x̂) = 0 ⇐⇒ Ax̂ = b

12 / 29

Gauss elimination and Cholesky decomposition

The original system Ax = b is transformed to an equivalentsystem Ux = b̂, where U is the upper triangular matrix.

The transformation can be represented by a lower triangularmatrix L, then A = LU (LU factorization), that is, LUx = b. Bydefining b̂ ≡ Ux , we easily solve Lb̂ = b and Ux = b̂ byapplying the back substitution twice.

The decomposition is not that simple in practice: pivotingneeded ⇒ permutation matrices involved.

If A is positive definite, aii 6= 0 because aii = eTi Aei > 0.

If A is s.p.d., then a unique Cholesky decomposition(factorization) A = LLT exists, where L is a lower triangularmatrix.

Sparse matrix: Not more than ≈ 5% nonzero entries.

Fill-in. Band matrix. Number of operations.

13 / 29

Condition number

Let ‖ · ‖ be an induced norm and let A is nonsingular. The value

κ(A) = ‖A‖ ‖A−1‖

is called the condition number of the matrix A with respect tothe norm ‖ · ‖.

If A is symmetric and positive definite and if we use ‖ · ‖2, thenκ(A) = λmax/λmin.

Let Ax0 = b0 and Ax1 = b1, where b0 6= b1, then

‖x1 − x0‖‖x0‖

≤ κ(A)‖b1 − b0‖

‖b0‖. (23)

Moreover, b0 6= 0 and b0 6= b1 exist such that (23) becomes theequality.

14 / 29

If the condition number is large, we say that the matrix isill-conditioned.

Ill-conditioned systems might be (and often are) difficult to solvewith the desired accuracy.

15 / 29

Hilbert matrix H(n)

H(n) = (aij), where aij =1

i + j − 1, i , j = 1,2, . . . ,n.

As an example take H(3) =

1 1/2 1/31/2 1/3 1/41/3 1/4 1/5

.

Let us solve H(n)x =

1...1

for n = 2,3, . . . ,14.

16 / 29

1 2−2

0

2

4

6

Exact solution. ||Ax−b||∞ = 0.00E+00

1 2

10−15.3

10−15.1

|exact solutioni − numer. solution

i| Cond. numb. = 1.9E+01 ||Ax−b||

∞ = 2.22E−16

17 / 29

1 2 3−40

−20

0

20

40


1 2 310

−15

10−14

10−13


i| Cond. numb. = 5.2E+02 ||Ax−b||

∞ = 8.88E−16

18 / 29

1 2 3 4−200

−100

0

100

200


1 2 3 410

−14

10−12

10−10


i| Cond. numb. = 1.6E+04 ||Ax−b||

∞ = 3.55E−15

19 / 29

1 2 3 4 5−2000

−1000

0

1000


1 2 3 4 510

−12

10−10

10−8


i| Cond. numb. = 4.8E+05 ||Ax−b||

∞ = 2.84E−14

20 / 29

1 2 3 4 5 6−1

−0.5

0

0.5

1x 10

4 Exact solution. ||Ax−b||∞ = 0.00E+00

1 2 3 4 5 610

−10

10−8

10−6


i| Cond. numb. = 1.5E+07 ||Ax−b||

∞ = 8.53E−14

21 / 29

1 2 3 4 5 6 7−4

−2

0

2

4x 10

4 Exact solution. ||Ax−b||∞ = 0.00E+00

1 2 3 4 5 6 710

−10

10−5

100


i| Cond. numb. = 4.8E+08 ||Ax−b||

∞ = 1.25E−12

22 / 29

1 2 3 4 5 6 7 8−2

0

2

4x 10

5Exact solution. ||Ax−b||

∞ = 0.00E+00

1 2 3 4 5 6 7 810

−10

10−5

100


i| Cond. numb. = 1.5E+10 ||Ax−b||

∞ = 3.64E−12

23 / 29

1 2 3 4 5 6 7 8 9−2

−1

0

1

2x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 910

−5

100

105


i| Cond. numb. = 4.9E+11 ||Ax−b||

∞ = 1.82E−11

24 / 29

1 2 3 4 5 6 7 8 9 10−1

−0.5

0

0.5

1x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 9 1010

−5

100

105


i| Cond. numb. = 1.6E+13 ||Ax−b||

∞ = 1.02E−10

25 / 29

1 2 3 4 5 6 7 8 9 10 11−5

0

5x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 9 10 1110

−5

100

105


i| Cond. numb. = 5.2E+14 ||Ax−b||

∞ = 9.31E−10

26 / 29

1 2 3 4 5 6 7 8 9 10 11 12−4

−2

0

2x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 9 10 11 1210

0

105

1010


i| Cond. numb. = 1.7E+16 ||Ax−b||

∞ = 2.79E−09

27 / 29

1 2 3 4 5 6 7 8 9 10 11 12 13−2

−1

0

1

2x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 9 10 11 12 1310

0

105

1010


i| Cond. numb. = 1.8E+18 ||Ax−b||

∞ = 2.51E−08

28 / 29

1 2 3 4 5 6 7 8 9 10 11 12 13 14−1

−0.5

0

0.5

1x 10


∞ = 0.00E+00

1 2 3 4 5 6 7 8 9 10 11 12 13 1410

0

105

1010


i| Cond. numb. = 3.1E+17 ||Ax−b||

∞ = 1.34E−07

29 / 29

code: 101mat4–101mt4b today’s topics normed linear (vector) space symmetric matrix … · 2020....

Documents