giansalvo exin cirrincione unité #5 décomposition en valeurs singulières (svd) valeurs...
TRANSCRIPT
![Page 1: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/1.jpg)
Giansalvo EXIN CirrincioneGiansalvo EXIN Cirrincione
unité #5unité #5
![Page 2: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/2.jpg)
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
021 n valeurs singulièresvaleurs singulières
aam1m1
aamnmn
aa1111 aa1n1n
UU
00
00
00 nn
11 00
VV
=
Full SVDFull SVDHA U V
![Page 3: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/3.jpg)
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
aam1m1
aamnmn
aa1111 aa1n1n
UU
00
00
00 nn
11 00
VV
=
Reduced SVDReduced SVDˆ ˆ HA U V aam1m1
aamnmn
aa1111 aa1n1n
UU 00 nn
11 00
VV
= ^̂
![Page 4: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/4.jpg)
Décomposition en valeurs singulières (Décomposition en valeurs singulières (SVDSVD))
Full SVDFull SVD
unitaire
unitaire
diagona le
m m
n n
m n
H
U
A U V
V
Reduced SVDReduced SVD
colonnes orthonormales
un
ˆ
ˆ
itaire
diagon
ˆ
al
ˆ
e
m n
n n
m
H
n
A U
V
V
U
ˆ ˆˆˆ ˆˆ ˆ ˆ ˆH
HH H 2H HH HA U V U U U V VVA V V
ˆ ˆˆ ˆ ˆ
ˆ ˆ ˆˆ ˆ ˆˆ
H11H H H 2 H H
H H H2 1 H
A A A A
V V
U V V V U
V
P
UU
V
UVU
ˆ ˆ HP UU
![Page 5: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/5.jpg)
Approximation au sens des moindres carrées
x
2
Given , , ,
find such that is minimized.
m n m
n
A m n b
x b Ax
Example: polynomial data fittingExample: polynomial data fitting
![Page 6: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/6.jpg)
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8-4
-3
-2
-1
0
1
2
f(x)
xi
yi
1,données : ,i i i m
x y
1
1
2
1
1
1
approximation
mi
( )
équations et inconnues ( ) approximation au s
interpolation
( )
ens
n (
) m
i ii
nj
nj
j
i i j i i
j
j
k
n m
f x y
f x x
n n
n m
f x y x y
des moindres carrés
Approximation au sens des moindres carrées
![Page 7: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/7.jpg)
discrete square wavediscrete square wave
interpolationinterpolation
m m = = n n = = 1111
least squaresleast squares
m m = = 11 11 , , nn = = 88
Approximation au sens des moindres carrées
![Page 8: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/8.jpg)
Posons le problème matriciellement
(1) ( 1) ( 1)1 2 1 1 1 1
(1) ( 1) ( 1)1 2 2 2 2 2
(1) ( 1) ( 1)1 2
1 2
... ...
... ...
... ...
j nj n
j nj n
j ni j i n i i
x x x f x
x x x f x
x x x f x
x
(1) ( 1) ( 1)... ...j nm j m n m mx x f x
1
1
1,
( )
pour ,
kj
jj
i i i m
f x x
x y
(1) ( 1) ( 1)11 1 1
(1) ( 1) ( 1)22 2 2
(1) ( 1) ( 1)
(1) ( 1) ( 1)
(1)
1
1) )
2
( ( 1
1 ... ...
1 ... ...
1 ... ...
1 ... ...
1
j n
j n
j nii i i
j nn n n
j nm m m
i
n
f xx x x
f xx x x
f xx x x
f xx x x
x x x
a
n
mf x
X f
Approximation au sens des moindres carrées
![Page 9: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/9.jpg)
2
1
1
2
1
1 1
1
1 1
1 1 1
* argmin ( )
min ( ) avec ( )
principe : 1,...,
2 0
( *)
n
0
mi ( )m
i ii
mj
i ii
m nj
j i ii j
m
j
nj
i i iij
J J x yf x y
x xx y x
JnJ j
J
1
1 1
n mj
i ii
y x
système linéaire de n équations et n inconnues
11 1 1
1
1
0
1
1
1
( )
1
1
kj
i i i j i ij
n
ni i i
nn n n
nm m m
e x x
e x x
e x x
e x x
e f x y x y
e X y
1
1
2
2 2
( ) 2
( )
( ) 0
i
nn
T
m
T T
y
y
y
X X X
y
J e X y
J X X y yJ
Approximation au sens des moindres carrées
erreur erreur d’approximationd’approximation
Matrice de Vandermonde (1735-1796)
![Page 10: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/10.jpg)
Approximation au sens des moindres carrées
2 2
1min
( )
2
min ( )
'( ) 2 '( ) 0
22
T
T T T T T
T
T T
J e X y
X y X y
X X X y y y
J
J X X y J
G h
G h
T TX X X y
Équations normalesÉquations normales
forme quadratiqueforme quadratique
![Page 11: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/11.jpg)
x
2
Given , , ,
find such that is minimi
zed.
m n m
n
A m n
Ax b
b
x b Ax
find the closest point in range to so that
the norm of the residual (residu) is minimized
mAx A b
r b Ax
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
![Page 12: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/12.jpg)
range(range(AA)) range nA
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
y = A xy = A x
r = b - A xb
b m
ranger A
1
1 1
H H
H H H H
P A A A A
Pb A A A A b A A A A Ax Ax
Pb Ax
0
normal equations
H
H H
A r
A Ax A b
The system is nonsingular iff A has full rank.
= Pb= Pb
![Page 13: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/13.jpg)
x
morthogonal projector that maps onto range
m m
A
Ax Pb P
1
1 1
H H
H H H H
P A A A A
Pb A A A A b A A A A Ax Ax
Pb Ax
0
normal equations
H
H H
A r
A Ax A b
The system is nonsingular iff A has full rank.
1
x1
pseudoinverse of
H H
H H
n m
x A bA A A b
A A A A
A
![Page 14: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/14.jpg)
Solution par les équations normales
factorisation de Cholesky
AHA est une matrice n x n hermitienne strictement définie positive
1. Form the matrix AAHHAA and the vector AAH H bb2. Compute the Cholesky factorization AAHHAA = = RRHHRR3. Solve the lower-triangular system RRHH w w = = AAH H bb for w4. Solve the upper-triangular system R xR x = = ww for x
3
2
3
2
computation of flops
computation
WORK
o
f
: fl
f
o
lop3
s
s
p3
H
H
A A mn
nR R
nmn
![Page 15: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/15.jpg)
Solution par la factorisation QR (Householder)
2 32WORK : flops
32 nmn
x xˆ ˆˆ ˆ , , m n n nA QR Q R reduced reduced QR factorizationQR factorization
1
ˆ ˆ
ˆ ˆ
ˆˆˆ ˆ
ˆ ˆ
H
H
H H
P QQ
y Pb QQ b Ax Q
A
x
x Q
R
Q RR b
1. Compute the reduced QR factorization ˆ ˆA QR
2. Compute the vector ˆ HQ b
3. Solve the upper-triangular system for xˆˆ HRx Q b
![Page 16: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/16.jpg)
Solution par la SVD
2 3WORK : 1 f o s2 1 l pnmn
1
ˆ ˆ
ˆ ˆ ˆ
ˆ ˆˆ
ˆ
ˆ
H
H H
H HH
P UU
y Pb UU b Ax U V x
V x U b A V U
1. Compute the reduced SVD ˆ ˆ HA U V
2. Compute the vector ˆ HU b
3. Solve the diagonal system for wˆˆ Hw U b
4. Set x Vw
For this cost is approximately the same as for QR factorization,
but for the SVD is more expensive
m n
m n
![Page 17: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/17.jpg)
Comparison of algorithms
� speed : normal equations� standard : QR factorization� A close to singular : SVD
DrawbacksDrawbacks• normal equationsnormal equations : not always stable in the presence of rounding errorsnot always stable in the presence of rounding errors• QR factoriz.QR factoriz.: less-than-ideal stability properties if less-than-ideal stability properties if AA is close to singular is close to singular• SVDSVD : expensive for expensive for mm nn
![Page 18: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/18.jpg)
Conditionnement et précision
rela
su
tive condition number
px
f x
xf x
p
su supx x
A x x Ax x
xA
A
A A
A x Ax
Ax x
x
x
x x
1
1
1
1
1
1
x b Ax
b x A b
A x A b
xA A A
b
bA A A
x
A A A
accuracy : relative erro
r
f x f x
f x
backward stability
accuracy of
a B
accurac
S algorithm
y
machine
f x f x
fxO
x
![Page 19: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/19.jpg)
Conditionnement du problème des moindres carrées
range(range(AA))y = A xy = A x
r = b - A xb
= Pb= Pb
x A b y Pb ˆ ˆ ˆ ˆH HP QQ UU AA
Données : A , b Solutions : x , y
12 2 2
2
A A A
arccos 2
2
y
b
closeness of the fit
2 2 2 2
2 2
A x A x
y Ax 2 2 22 2 2 2
2 2 2
A AA AA xA
Ax Ax Ax
b Ax
0 11 A2A
![Page 20: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/20.jpg)
Conditionnement du problème des moindres carrées
Données : A , b Solutions : x , y
12 2 2
2
A A A
arccos 2
2
y
b
2 2 2 2
2 2
A x A x
y Ax
0 11 A2A
2-norm relative condition numbers
exact for certain exact for certain b b
upper boundsupper bounds
![Page 21: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/21.jpg)
Stabilité des méthodes des moindres carrées
exemple
Least squares fitting of the function exp(sin(4)) on the interval [0,1] by a polynomial of degree 14
x15 = 1
highly ill-conditioned basis
very close fit
16machine 10
![Page 22: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/22.jpg)
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
The rounding errors have been amplified by a factor of order 10 9. This inaccuracy is explained by ill-conditioning, not instability.
reducedreduced
![Page 23: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/23.jpg)
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
implicit calculation of the product QH b
explicit formingQ
same accuracy
![Page 24: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/24.jpg)
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
7rel. err. 3 10
implicit calculation of the product QH b
explicit formingQ
same accuracy\ QR factorization
with column pivoting
ˆ ˆAP
b
R
A
Q
an order of magn. more acc.
![Page 25: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/25.jpg)
Stabilité des méthodes des moindres carrées
exemplefactorisation QR (Householder)
16machine 10
backward stable backward stable
This is true whether is computed via
ˆformation of or . It also holds for
House
expli
holde
mi
cit
implicitly
column pivot
r triangulari
n
za
in
tion with arbitr
.g
a y
ˆ
r
machine
H
AA A x
Q b
A
Q
b O
![Page 26: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/26.jpg)
Stabilité des méthodes des moindres carrées
exemple
SVD
16machine 10
backward stable backward stable
min machine
AA A x b O
A
It beats Householder triangularization with column It beats Householder triangularization with column pivoting ( pivoting ( MATLAB's \MATLAB's \ ) by a factor of about ) by a factor of about 33
![Page 27: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/27.jpg)
Stabilité des méthodes des moindres carrées
exemple
équations normales
16machine 10
unstable unstable
not even a single digit of accuracynot even a single digit of accuracy
factorisation de Cholesky
![Page 28: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/28.jpg)
Stabilité des méthodes des moindres carrées
BS least squares algorithmBS least squares algorithm
2 tanmachine
x xO
x
suppose ill-conditioned, i.e. 1, and is bounded away from 2A
and
ta 1
n
or
tan
0
2machineO machineO
The condition The condition number of the LS number of the LS problem may lie problem may lie anywhere in the anywhere in the range range to to 22 . .
![Page 29: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/29.jpg)
Stabilité des méthodes des moindres carrées
BS least squares algorithmBS least squares algorithm Cholesky factorization (BS)Cholesky factorization (BS)
2 tanmachine
x xO
x
2machine
x xO
x
suppose ill-conditioned, i.e. 1, and is bounded away from 2A
and
ta 1
n
or
tan
0
2machineO machineO 2
machineO 2machineO
2cond HA A
The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.
![Page 30: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/30.jpg)
Stabilité des méthodes des moindres carrées
The normal equations are typically unstable for The normal equations are typically unstable for ill-conditioned problems involving close fits.ill-conditioned problems involving close fits.
The solution of the full-rank least squares problem via the normal equations is unstable. Stability can be achieved, however, by restriction to a class of problems in which (A) is uniformly bounded above or (tan)/ is uniformly bounded below.
![Page 31: Giansalvo EXIN Cirrincione unité #5 Décomposition en valeurs singulières (SVD) valeurs singulières a m1 a mn a 11 a 1n U 0 0 0 n n 10V = Full SVD](https://reader036.vdocuments.pub/reader036/viewer/2022062511/55153431550346c77d8b5a13/html5/thumbnails/31.jpg)
FINEFINE