Chapter 6 – Inner Product Spaces

Per-Olof Perssonpersson@berkeley.edu

Department of MathematicsUniversity of California, Berkeley

Math 110 Linear Algebra

Inner Products

Definition

An inner product on a vector space V over F is a function thatassigns a scalar 〈x, y〉 for every x, y ∈ V, such that for allx, y, z ∈ V and c ∈ F :

(a) 〈x+ z, y〉 = 〈x, y〉+ 〈z, y〉(b) 〈cx, y〉 = c〈x, y〉(c) 〈x, y〉 = 〈y, x〉 (complex conjugation)

(d) 〈x, x〉 > 0 if x 6= 0

Example

For x = (a1, . . . , an) and y = (b1, . . . , bn) in Fn, the standard innerproduct is defined by

〈x, y〉 =n∑i=1

aibi

Inner Products

Example

For f, g ∈ V = C([0, 1]), an inner product is given by〈f, g〉 =

∫ 10 f(t)g(t) dt.

Definition

The conjugate transpose or adjoint of A ∈ Mm×n(F ) is the n×mmatrix A∗ such that (A∗)ij = Aji for all i, j.

Example

The Frobenius inner product on V = Mn×n(F ) is defined by〈A,B〉 = tr(B∗A) for A,B ∈ V.

Properties of Inner Product Spaces

A vector space V over F with a specific inner product is called aninner product space. If F = C, V is a complex inner productspace, and if F = R, V is a real inner product space.

Theorem 6.1

For an inner product space V, x, y, z ∈ V, and c ∈ F :

(a) 〈x, y + z〉 = 〈x, y〉+ 〈x, z〉(b) 〈x, cy〉 = c〈x, y〉(c) 〈x, 0〉 = 〈0, x〉 = 0

(d) 〈x, x〉 = 0 if and only if x = 0

(e) If 〈x, y〉 = 〈x, z〉 for all x ∈ V, then y = z

Norms

Definition

Let V be an inner product space. For x ∈ V, the norm or thelength of x is ‖x‖ =

√〈x, x〉.

Example

For x = (a1, . . . , an) ∈ V = Fn, the Euclidean length is the norm

‖x‖ =

[n∑i=1

|ai|2]1/2

Theorem 6.2

For an inner product space V over F and all x, y ∈ V, c ∈ F :

(a) ‖cx‖ = |c| · ‖x‖(b) ‖x‖ = 0 if and only if x = 0. In any case, ‖x‖ ≥ 0.

(c) (Cauchy-Schwarz Inequality) |〈x, y〉| ≤ ‖x‖ · ‖y‖(d) (Triangle Inequality) ‖x+ y‖ ≤ ‖x‖+ ‖y‖

Orthogonality

Definition

Let V be an inner product space. Vectors x, y ∈ V are orthogonal(perpendicular) if 〈x, y〉 = 0. A subset S of V is orthogonal if anytwo distinct vectors in S are orthogonal. A vector x ∈ V is a unitvector if ‖x‖ = 1. A subset S of V is orthonormal if S isorthogonal and consists entirely of unit vectors.

Example

Consider the inner product space H of continuous complex-valuedfunctions defined on [0, 2π] with the inner product

〈f, g〉 = 1

2π

∫ 2π

0f(t)g(t) dt.

Let fn(t) = eint for any integer n, where 0 ≤ t ≤ 2π. ThenS = {fn : n is an integer} is an orthonormal subset of H.

Orthonormal Bases

Definition

A subset of an inner product space V is an orthonormal basis for Vif it is an ordered basis that is orthonormal.

Theorem 6.3

Let V be an inner product space and S = {v1, . . . , vk} anorthogonal subset of V consisting of nonzero vectors. If

y ∈ span(S), then y =

k∑i=1

〈y, vi〉‖vi‖2

vi.

Corollary 1

If in addition S is orthonormal, then y =∑k

i=1〈y, vi〉vi.

Corollary 2

Let V be an inner product space, and S an orthogonal subset of Vconsisting of nonzero vectors. Then S is linearly independent.

Gram-Schmidt Orthogonalization

Theorem 6.4

Let V be an inner product space and S = {w1, . . . , wn} a linearlyindependent subset of V. Define S′ = {v1, . . . , vn}, where v1 = w1

and

vk = wk −k−1∑j=1

〈wk, vj〉‖vj‖2

vj for 2 ≤ k ≤ n.

Then S′ is an orthogonal set of nonzero vectors such thatspan(S′) = span(S).

This construction of {v1, . . . , vn} is called the Gram-Schmidtprocess.

Representations in Orthonormal Bases

Theorem 6.5

Let V be a nonzero finite-dimensional inner product space. Then Vhas an orthonormal basis β. Furthermore, if β = {v1, . . . , vn} andx ∈ V, then

x =

n∑i=1

〈x, vi〉vi.

Corollary

Let V be a finite-dimensional inner product space with anorthonormal basis β = {v1, . . . , vn}. Let T be a linear operator onV, and let A = [T]β. Then for any i and j, Aij = 〈T(vj), vi〉.

Definition

Let β be an orthonormal subset (possibly infinite) of an innerproduct space V, and let x ∈ V. The Fourier coefficients of xrelative to β are the scalars 〈x, y〉, where y ∈ β.

Orthogonal Complement

Definition

Let S be a nonempty subset of an inner product space V. Wedefine S⊥ to be the set of all vectors in V that are orthogonal toevery vector in S; that is, S⊥ = {x ∈ V : 〈x, y〉 = 0 for all y ∈ S}.The set S⊥ is called the orthogonal complement of S.

Theorem 6.6

Let W be a finite-dimensional subspace of an inner product spaceV, and let y ∈ V. Then there exist unique vectors u ∈W andz ∈W⊥ such that y = u+ z. Furthermore, if {v1, . . . , vk} is anorthonormal basis for W, then u =

∑ki=1〈y, vi〉vi.

Corollary

The vector u in Thm 6.6 is the unique vector in W that is “closest”to y; that is, for any x ∈W, ‖y− x‖ ≥ ‖y− u‖, and this inequalityis an equality if and only if x = u (the orthogonal projection).

Orthogonal Extension

Theorem 6.7

Suppose that S = {v1, . . . , vk} is an orthonormal set in ann-dimensional inner product space V. Then

(a) S can be extended to an orthonormal basis{v1, . . . , vk, vk+1, . . . , vn} for V.

(b) If W = span(S), then S1 = {vk+1, . . . , vn} is an orthonormalbasis for W⊥.

(c) If W is any subspace of V, thendim(V) = dim(W) + dim(W⊥).

The Adjoint

Theorem 6.8

Let V be a finite-dimensional inner product space over F , and letg : V→ F be a linear transformation. Then there exists a uniquevector y ∈ V such that g(x) = 〈x, y〉 for all x ∈ V.

Theorem 6.9

Let V be a finite-dimensional inner product space, and let T be alinear operator on V. Then there exists a unique functionT∗ : V→ V such that 〈T(x), y〉 = 〈x,T∗(y)〉 for all x, y ∈ V.Furthermore, T∗ is linear.

The linear operator T∗ is called the adjoint of T.

Properties of Adjoints

Theorem 6.10

Let V be a finite-dimensional inner product space, and let β be anorthonormal basis for V. If T is a linear operator on V, then

[T∗]β = [T]∗β.

Corollary

Let A be an n× n matrix. Then LA∗ = (LA)∗.

Theorem 6.11

Let V be an inner product space, and T,U linear operators on V.

(a) (T + U)∗ = T∗ + U∗

(b) (cT)∗ = cT∗ for any c ∈ F(c) (TU)∗ = U∗T∗

(d) T∗∗ = T

(e) I∗ = I

Properties of Adjoints

Corollary

Let A,B be n× n matrices. Then

(a) (A+B)∗ = A∗ +B∗

(b) (cA)∗ = cA∗ for all c ∈ F(c) (AB)∗ = B∗A∗

(d) A∗∗ = A

(e) I∗ = I

Normal Operators

Lemma

Let T be a linear operator on a finite-dimensional inner productspace V. If T has en eigenvector, then so does T∗.

Theorem 6.14 (Schur)

With T as in the lemma, suppose that the characteristicpolynomial of T splits. Then there exists an orthonormal basis βfor V such that the matrix [T]β is upper triangular.

Definition

Let V be an inner product space and T a linear operator on V. T isnormal if TT∗ = T∗T. An n× n real or complex matrix A isnormal if AA∗ = A∗A.

Properties of Normal Operators

Theorem 6.15

With V an inner product space and T a normal operator on V:

(a) ‖T(x)‖ = ‖T∗(x)‖ for all x ∈ V.

(b) T− cI is normal for every c ∈ F .

(c) If x is an eigenvector of T, then x is also an eigenvector ofT∗. In fact, if T(x) = λx, then T∗(x) = λx.

(d) If λ1, λ2 are distinct eigenvalues of T then the correspondingeigenvectors x1, x2 are orthogonal.

Theorem 6.16

Let T be a linear operator on a finite-dimensional complex innerproduct space V. Then T is normal if and only if there exists anorthonormal basis for V consisting of eigenvectors of T.

Self-Adjoint Operators

Definition

Let T be a linear operator on an inner product space V. T isself-adjoint (Hermitian) if T = T∗. An n× n real or complexmatrix A is self-adjoint (Hermitian) if A = A∗.

Lemma

Let T be a self-adjoint operator on a finite-dimensional innerproduct space V. Then

(a) Every eigenvalue of T is real.

(b) Suppose that V is a real inner product space. Then thecharacteristic polynomial of T splits.

Theorem 6.17

Let T be a linear operator on a finite-dimensional real innerproduct space V. Then T is self-adjoint if and only if there existsan orthonormal basis β for V consisting of eigenvectors of T.

Unitary and Orthogonal Operators

Definition

Let T be a linear operator on a finite-dimensional inner productspace V over F . If ‖T(x)‖ = ‖x‖ for all x ∈ V, we call T a unitaryoperator if F = C and an orthogonal operator if F = R.

Theorem 6.18

Let T be a linear operator on a finite-dimensional inner productspace V. Then the following statements are equivalent.

(a) TT∗ = T∗T = I

(b) 〈T(x),T(y)〉 = 〈x, y〉 for all x, y ∈ V

(c) If β is an orthonormal basis for V, then T(β) is anorthonormal basis for V.

(d) There exists an orthonormal basis β for V such that T(β) isan orthonormal basis for V.

(e) ‖T(x)‖ = ‖x‖ for all x ∈ V.

Unitary and Orthogonal Operators

Lemma

Let U be a self-adjoint operator on a finite-dimensional innerproduct space V. If 〈x,U(x)〉 = 0 for all x ∈ V, then U = T0.

Corollary 1

Let T be a linear operator on a finite-dimensional real innerproduct space V. Then V has an orthonormal basis of eigenvectorsof T with eigenvalues of absolute value 1 if and only if T is bothself-adjoint and orthogonal.

Corollary 2

Let T be a linear operator on a finite-dimensional complex innerproduct space V. Then V has an orthonormal basis of eigenvectorsof T with eigenvalues of absolute value 1 if and only if T is unitary.

Orthogonal and Unitary Matrices

Definition

A square matrix A is called an orthogonal matrix ifAtA = AAt = I and unitary if A∗A = AA∗ = I.

Two matrices A,B are unitarily [orthogonally] equivalent if andonly if there exists a unitary [orthogonal] matrix P such thatA = P ∗BP .

Theorem 6.19

Let A be a complex n× n matrix. Then A is normal if and only ifA is unitarily equivalent to a diagonal matrix.

Theorem 6.20

Let A be a real n× n matrix. Then A is symmetric if and only ifA is orthogonally equivalent to a real diagonal matrix.

Schur and Unitarily/Orthogonally Equivalent Matrices

Theorem 6.21 (Schur)

Let A ∈ Mn×n(F ) be a matrix whose characteristic polynomialsplits over F .

(a) If F = C, then A is unitarily equivalent to a complex uppertriangular matrix.

(b) If F = R, then A is orthogonally equivalent to a real uppertriangular matrix.