representation learning using multi-task deep neural networksfor semantic classification and...
TRANSCRIPT
![Page 1: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/1.jpg)
Komachi Lab
M1 宮崎亮輔
2015/06/24
Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval !Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, Ye-Yi Wang
NAACL読み会2015
※このスライド中の図はこの論文中のものです
![Page 2: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/2.jpg)
Komachi Lab
Abstract✤ 最近のDNNって目的のタスクに対して直接最適化してなかったり!
✤ 任意のタスクへの教師ありもトレーニングデータの不足とかあるし!
✤ ということでMulti-Task DNN for Representation を提案!
- C&Wとの違い?→今回は処理の違うタスク同士!
- query classification(今回は4ドメイン)とranking for web search!
✤ データ量が増えるだけでなく、正則化の効果も(ドメインアダプテーション)
2
![Page 3: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/3.jpg)
Komachi Lab
Architecture
3
![Page 4: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/4.jpg)
Komachi Lab
Input
✤ 入力はクエリ or ドキュメント (Bag-of-Words 500k次元)!
- つまり語彙数500kのOne hot vector
4
![Page 5: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/5.jpg)
Komachi Lab
Word Hash Layer
✤ 次の層で文字trigramの空間にmapする(50k次元)!
- 未知語の問題が解消!
- 同単語の複数のspelingが近くにmapされる
5
※単語境界は”#”で表現
![Page 6: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/6.jpg)
Komachi Lab
Semantic Representation Layer
✤ 意味表現(300次元)!
- l2 = f(W1・l1), f() = tanh
6
![Page 7: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/7.jpg)
Komachi Lab
Task-Specific Layer
7
✤ タスク固有中間表現(128次元)!
- l3 = f(Wt2・l2) , t = task!
- 入力がクエリ: l3 = Q, ドキュメント: l3 = D
クエリ分類タスク:
![Page 8: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/8.jpg)
Komachi Lab
Task-Specific Layer
8
,クエリ分類クエリ分類
✤ タスク固有中間表現(128次元)!
- l3 = f(Wt2・l2) , t = task!
- 入力がクエリ: l3 = Q, ドキュメント: l3 = D
タスク:
![Page 9: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/9.jpg)
Komachi Lab
Task-Specific Layer
9
,web searchクエリ分類 ,クエリ分類
✤ タスク固有中間表現(128次元)!
- l3 = f(Wt2・l2) , t = task!
- 入力がクエリ: l3 = Q, ドキュメント: l3 = D
タスク:
![Page 10: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/10.jpg)
Komachi Lab
Task-Specific Layer
10
,web search,web searchクエリ分類 ,クエリ分類
✤ タスク固有中間表現(128次元)!
- l3 = f(Wt2・l2) , t = task!
- 入力がクエリ: l3 = Q, ドキュメント: l3 = D
タスク:
![Page 11: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/11.jpg)
Komachi Lab
Task-Specific Layer
11
,web search,web searchクエリ分類 ,クエリ分類 ,web search
✤ タスク固有中間表現(128次元)!
- l3 = f(Wt2・l2) , t = task!
- 入力がクエリ: l3 = Q, ドキュメント: l3 = D
タスク:
![Page 12: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/12.jpg)
Komachi Lab
Task-Specific Layer
12
✤ クエリ分類(ロジスティック回帰)!
- 二値分類(対応するドメインに属すか否か)!
- 一つのクエリは複数のドメインに属すことができる!
- ドメインの数だけ分類器,拡張性がある
g()はシグモイド関数
![Page 13: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/13.jpg)
Komachi Lab
Task-Specific Layer
13
✤ クエリ分類(ロジスティック回帰)!
- 二値分類(対応するドメインに属すか否か)!
- 一つのクエリは複数のドメインに属すことができる!
- ドメインの数だけ分類器,拡張性がある
![Page 14: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/14.jpg)
Komachi Lab
Task-Specific Layer
14
✤ Web検索!
- クエリとのコサイン類似度→softmax!
- P(D|Q)の降順でランキング γはハイパーパラメーター
![Page 15: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/15.jpg)
Komachi Lab
Architecture
15
![Page 16: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/16.jpg)
Komachi Lab
Training✤ ミニバッチSGD!
✤ クエリ分類: 式(5)!
- クロスエントロピーロス最小化!
❖ Web検索: 式(6)!
- 負の対数尤度最小化!
❖ 初期化には以下の範囲から一様分布
16
※800k iterations, 13hours
![Page 17: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/17.jpg)
Komachi Lab
Experimental Data Sets
✤ クエリ分類には商用検索エンジンのログ1年分(人手のラベル付き)!
✤ Web検索は12,071のクエリを含みクエリとドキュメントの組み合わせに5段階の関連度
17
![Page 18: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/18.jpg)
Komachi Lab
Query Classification✤ クエリ分類のベースライン!
- SVM-word unigram, bigram, trigram, surface!
- SVM-letter 文字trigram!
- DNN マルチタスクではないDNN
18
![Page 19: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/19.jpg)
Komachi Lab
Query Classification Results
✤ SVM < DNN :意味表現が重要!
✤ DNN < MT-DNN :マルチタスクは有用
19
※評価はAUC
![Page 20: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/20.jpg)
Komachi Lab
Web Search
✤ Web検索のベースライン!
- 一般的なベースラインTF-IDF, LDA, etc.!
- DSSMマルチタスクではないDNN
20
![Page 21: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/21.jpg)
Komachi Lab
Web Search Results
✤ State-of-the-art(DSSM)を超えたやはりマルチタスクは有用
21
※NDCGはrankingを評価する指標
![Page 22: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/22.jpg)
Komachi Lab
Domain Adaptation✤ クエリ分類のひとつのドメインを除いてMT-DNNで学習!
- 学習したSemantic Representationを素性にSVMで分類!
- ベースラインはSVM-Word, SVM-letter
22
✤ 一般的なFeed Forward DNNでも比較!
- Semantic RepresentationをW1にしてW2, Wt3を学習!
- W1をランダムに初期化, W1, W2, Wt3を学習!
- SVM-Word
![Page 23: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/23.jpg)
Komachi Lab 23
![Page 24: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/24.jpg)
Komachi Lab
Domain Adaptation✤ クエリ分類のひとつのドメインを除いてMT-DNNで学習!
- 学習したSemantic Representationを素性にSVMで分類!
- ベースラインはSVM-Word, SVM-letter
24
✤ 一般的なFeed Forward DNNでも比較!
- Semantic RepresentationをW1にしてW2, Wt3を学習!
- W1をランダムに初期化, W1, W2, Wt3を学習!
- SVM-Word
![Page 25: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/25.jpg)
Komachi Lab 25
![Page 26: Representation Learning Using Multi-Task Deep Neural Networksfor Semantic Classification and Information Retrieval](https://reader033.vdocuments.pub/reader033/viewer/2022042818/55c0f22abb61eb8f428b45c5/html5/thumbnails/26.jpg)
Komachi Lab
Conclusion
✤ 分類とランキングという異なるタスクを合わせて
DNNでのマルチタスク学習を提案!
✤ ベースラインを上回り、Web検索ではState-of-the-art!
✤ マルチタスク学習によりドメインアダプテーションされた表現を学習できた
26