linear algebra background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...net ix...
TRANSCRIPT
![Page 1: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/1.jpg)
Linear Algebra Background
Dan Sheldon
September 16, 2014
![Page 2: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/2.jpg)
Motivation
Multivariate linear regression:
y(1) ≈ θ0 + θ1x(1)1 + θ2x
(1)2 + . . .+ θnx
(1)n
y(2) ≈ θ0 + θ1x(2)1 + θ2x
(2)2 + . . .+ θnx
(2)n
. . .
y(m) ≈ θ0 + θ1x(m)1 + θ2x
(m)2 + . . .+ θnx
(m)n
After linear algebray ≈ Xθ
![Page 3: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/3.jpg)
Motivation
Multivariate linear regression:
y(1) ≈ θ0 + θ1x(1)1 + θ2x
(1)2 + . . .+ θnx
(1)n
y(2) ≈ θ0 + θ1x(2)1 + θ2x
(2)2 + . . .+ θnx
(2)n
. . .
y(m) ≈ θ0 + θ1x(m)1 + θ2x
(m)2 + . . .+ θnx
(m)n
After linear algebray ≈ Xθ
![Page 4: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/4.jpg)
Linear Algebra in ML
Linear Algebra
I Succinct notation for models and algorithms
I Numerical tools (save coding!)
θ = (XTX)−1XTy
I Inspiration for new models and problems: Netflix
![Page 5: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/5.jpg)
Netflix Movie Recommendations
Gladiator Silence of the Lambs WALL-E Toy Story
Alice 5 4 1Bob 5 2
Carol 5David 5 5Eve 5 4
Matrix completion problem, matrix factorization
![Page 6: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/6.jpg)
Topics
I Matrices
I Vectors
I Matrix-Matrix multiplication (and special cases)
I Tranpose
I Inverse
![Page 7: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/7.jpg)
Matrices
I A matrix is an rectangular array of numbers
A =
101 1054 1310 47
I When A has m rows and n columns, we say that:
I A is an m× n matrixI A ∈ Rm×n
I The entry in row i and column j is denoted Aij
I sometimes aij or (A)ij
![Page 8: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/8.jpg)
Matrices
Example
A =
101 1054 1310 47
I A ∈ R3×2
I A11 = 101
I A32 =
I A22 =
I A23 =
![Page 9: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/9.jpg)
Vectors
I A vector is an n× 1 matrix:
x =
82.41−10
I We write x ∈ Rn (instead of x ∈ Rn×1)
I The ith entry is xi
![Page 10: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/10.jpg)
Vectors
Example
x =
82.41−10
I x ∈ R4
I x1 =
I x4 =
![Page 11: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/11.jpg)
Addition
I If two matrices have the same size, we can add them byadding corresponding elements[
1 23 4
]+
[3 5−1 0
]=
[4 72 4
]I Subtraction is similar
I Matrices of different sizes cannot be added or subtracted
![Page 12: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/12.jpg)
Scalar Multiplication
I A scalar x ∈ R is a real number (i.e., not a vector)
e.g., 2, 3, π,√2, 1.843, . . .
I Scalar times matrix:
2 ·[1 3−2 0
]=
[2 6−4 0
](multiply each entry by the scalar)
![Page 13: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/13.jpg)
Matrix-Matrix Multiplication
I Can multiply two matrices if their inner dimensions match
A ∈ Rm×n, B ∈ Rn×p
C = AB ∈ Rm×p
I The product has entries
Cij =
n∑k=1
AikBkj
![Page 14: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/14.jpg)
Matrix-Matrix Multiplication
Cij =
n∑k=1
AikBkj
Move along ith row of A and jth row of B. Multiplycorresponding entries, then add.
c11 c12 c13c21 c22 c23c31 c32 c33c41 c42 c43
=
a11 a12a21 a22a31 a32a41 a42
[b11 b12 b13b21 b22 b23
]
c32 = a31b12 + a32b22
![Page 15: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/15.jpg)
Matrix-Matrix Multiplication
Example
A =
[1 −10 3
], B =
[3 2−1 0
]
AB =
[1 −10 3
] [3 2−1 0
]=
[4 2−3 0
]
![Page 16: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/16.jpg)
Matrix-Matrix Multiplication
Example
A =
[1 −10 3
], B =
[3 2−1 0
]
AB =
[1 −10 3
] [3 2−1 0
]=
[4 2−3 0
]
![Page 17: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/17.jpg)
Multiplication Properties
I Associative(AB)C = A(BC)
I DistributiveA(B + C) = AB +AC
(B + C)D = BD + CD
I Not commutativeAB 6= BA
![Page 18: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/18.jpg)
Matrix-Vector Multiplication
A (worthy) special case of matrix-matrix multiplication:
A ∈ Rm×n, x ∈ Rn
y = Ax ∈ Rm
Definition
yi =
n∑j=1
Aijxj
![Page 19: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/19.jpg)
Matrix-Vector Multiplication
yi =
n∑j=1
Aijxj
y1y2y3y4
=
a11 a12a21 a22a31 a32a41 a42
[x1x2]
y3 = a31x1 + a32x2
![Page 20: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/20.jpg)
Matrix-Vector Multiplication
Example
A =
[1 −10 3
], x =
[1−1
]z =
[81.5
]
I Ax =
[1 −10 3
] [1−1
]=
[2−3
]I Az =
[6.54.5
]
![Page 21: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/21.jpg)
Matrix-Vector Multiplication
Example
A =
[1 −10 3
], x =
[1−1
]z =
[81.5
]
I Ax =
[1 −10 3
] [1−1
]=
[2−3
]
I Az =
[6.54.5
]
![Page 22: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/22.jpg)
Matrix-Vector Multiplication
Example
A =
[1 −10 3
], x =
[1−1
]z =
[81.5
]
I Ax =
[1 −10 3
] [1−1
]=
[2−3
]I Az =
[6.54.5
]
![Page 23: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/23.jpg)
Matrix-Vector Multiplication
Example
A =
[1 −10 3
], x =
[1−1
]z =
[81.5
]
I Ax =
[1 −10 3
] [1−1
]=
[2−3
]I Az =
[6.54.5
]
![Page 24: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/24.jpg)
Transpose
Transposition of a matrix swaps the rows and columns
A =
[1 −10 3
], AT =
[1 0−1 3
].
Definition:
I Let A ∈ Rm×n
I The transpose AT ∈ Rn×m has entries
(AT )ij = Aji.
![Page 25: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/25.jpg)
Transpose
Example
A =
3 2−1 01 4
AT =
[3 −1 12 0 4
]
Example
x =
1−32
xT =[1 −3 2
]
![Page 26: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/26.jpg)
Transpose
Example
A =
3 2−1 01 4
AT =
[3 −1 12 0 4
]
Example
x =
1−32
xT =[1 −3 2
]
![Page 27: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/27.jpg)
Transpose
Example
A =
3 2−1 01 4
AT =
[3 −1 12 0 4
]
Example
x =
1−32
xT =
[1 −3 2
]
![Page 28: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/28.jpg)
Transpose
Example
A =
3 2−1 01 4
AT =
[3 −1 12 0 4
]
Example
x =
1−32
xT =[1 −3 2
]
![Page 29: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/29.jpg)
Dot product
I A special special-case of matrix-matrix multiplication
I Let x,y be vectors of same size (x,y ∈ Rn).
I Their dot product is
xTy =
n∑i=1
xiyi
=[x1 x2 . . . xn
]y1y2...yn
![Page 30: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/30.jpg)
Vector Norm
I The norm of a vector
‖x‖ =√x21 + x22 + . . .+ x2n
=√xTx
I Geometric interpretation: length of the vector
![Page 31: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/31.jpg)
Transpose Properties
I Transpose of transpose
(AT )T = A
I Transpose of sum
(A+B)T = AT +BT
I Transpose of product
(AB)T = BTAT
![Page 32: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/32.jpg)
Identity
I The identity matrix I ∈ Rn×n has entries
Iij =
{1 i = j
0 i 6= j,
I1×1 = [1], I2×2 =
[1 00 1
], I3×3 =
1 0 00 1 00 0 1
.I For any A,B of appropriate dimensions
IA = A
BI = B
![Page 33: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/33.jpg)
Inverse
I The inverse A−1 ∈ Rn×n of a square matrix A ∈ Rn×n
satisfiesAA−1 = I = A−1A
I Compare to division of scalars
xx−1 = 1 = x−1x
I Not all matrices are invertible
I E.g., A not square, A = [0], A =
[0 00 0
], many more
![Page 34: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/34.jpg)
Inverse
Example
A =
[1 00 2
], B =
[1 00 1
2
]Is B the inverse of A?
Example
A =
[1 −10 3
]A−1 =
[1 1
30 1
3
]Verify on your own.
![Page 35: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/35.jpg)
Inverse
Example
A =
[1 00 2
], B =
[1 00 1
2
]Is B the inverse of A?
Example
A =
[1 −10 3
]A−1 =
[1 1
30 1
3
]Verify on your own.
![Page 36: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/36.jpg)
Inverse Properties
I Inverse of inverse(A−1)−1 = A
I Inverse of product
(AB)−1 = B−1A−1
I Inverse of transpose
(A−1)T = (AT )−1 := A−T
![Page 37: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/37.jpg)
In MATLAB
See demo.m
![Page 38: Linear Algebra Background - people.cs.umass.edusheldon/teaching/mhc/cs335/2014fa/lec/05...Net ix Movie Recommendations Gladiator Silence of the Lambs WALL-E Toy Story Alice 5 4 1 Bob](https://reader030.vdocuments.pub/reader030/viewer/2022041208/5d65389388c9936d238b5709/html5/thumbnails/38.jpg)
What You Should Know
I Definitions of matrices and vectorsI Meaning of matrix multiplication
I Systems of equations −→ matrix-vector equations
I Properties of multiplicationI Properties of inverse, transpose
I Get familiar with these as course goes on