1. 1. : NAIST Computational Linguistic Lab D1 Masayoshi Kondo 5 -
2. 2. 00: Deep Learning : 165 ()
3. 3. XX: () . . .
4. 4. 00: (CNN) (RNN)
5. 5. 00: () - (Deep Learning) / ()
6. 6. 00: (3) - (overtting) .
7. 7. () () () 00: (4) - !!
8. 8. 1. 2. 3. 4.(pre-training) . 00: (autoencoder)
9. 9. 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7 () ()!!
10. 10. 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7
11. 11. (normalization)(3.6.1) 01: (whitening) . Analysis of single-layer networks in supervised feature learning. [A.Coates+, AISTATS2011] . 5.5
12. 12. DDP() ( P ) un = Pxn (n =1,!,n) U = 1 N unun T = 1 N UUT n=1 N () P U 02: 5.5 x =[x1, x2,!, xD ]X =[x1,x2,!,xN ] DN X = 1 N xnxn T = 1 N XXT n=1 N
13. 13. ZCA (zero-phase whitening,/ZCA: zero-phase component analysis,) PZCA = ED - 1 2 ET or D-1/2 Q D E . X X P 03: 5.5 PCA (PCA: principal component analysis, ) Ppca = QD - 1 2 ET PZCA = E(D+I) - 1 2 ET
14. 14. 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7
15. 15. (pre-training) . . 1. 1. 2. . 04: 5.6
16. 16. 05: (stacked autoencoder) ( Input ) (Output) Xn Zn (3) Zn (2) Zn (2) Zn (3) W(2) W(3) W(4) W(2) W(3) W(4) W(2) W(3) W(4) W(5)
17. 17. 06: (stacked autoencoder) 1. . 2. . 3. 1 . 4. . . . . SVM. xzx.
18. 18. 5 5.1 5.2 5.3 5.4 5.5 5.6 5.7
19. 19. 07: 5.7.1 (deep autoencoder) . . W(2) W(3) W(4) y W(2) W(3) W(4) W(2)W(3)W(4) x x x (deep autoencoder)
20. 20. 08: 5.7.2 (denoising autoencoder) (RBM). . 1. x(). x . 2. x. 3. . x 2 x x ~ N(0, 2 I) x = x +x x x( x) = f ( Wf (Wx + b)+ b)
21. 21. 09: 5.7.2 (denoising autoencoder) . . . x . (). . 1. () 2. 3. & . X. X. .
22. 22.