a neural attention model for sentence summarization [rush+2015]
TRANSCRIPT
![Page 1: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/1.jpg)
2015/10/24 EMNLP2015読み会@PFI
kiyukuta
![Page 2: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/2.jpg)
“文の要約” 文を短くする(言い換えとかも含めて) !
![Page 3: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/3.jpg)
“文の要約” 文を短くする(言い換えとかも含めて) !
≠ Document Summarization 文書から短い文書を作る
≠ Sentence Compression 文から単語を削除して短くする
語順の入れ替えも無し
![Page 4: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/4.jpg)
headline generationやvery short summary とか言われるタスクとほぼおなじ
NN機械翻訳で話題のAttentionモデルを移植 (ただし,各コンポーネントを簡易化している)
背景
This
(3.2節の最後)
機械翻訳からインスパイアされた手法が以前から存在 +
最近はNeural Networkベースの機械翻訳が盛ん
![Page 5: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/5.jpg)
提案手法
まず論文の図を使ってざっくり説明 そのあと式を使って説明
![Page 6: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/6.jpg)
Figure1. 提案手法の処理(終了時)の例
論文にある実例でざっくりイメージをつかむ
![Page 7: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/7.jpg)
…
![Page 8: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/8.jpg)
…
![Page 9: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/9.jpg)
…
?
![Page 10: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/10.jpg)
…
?結論からいうと 入力単語ベクトルを荷重平均して使うときの荷重
![Page 11: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/11.jpg)
途中状態で説明
![Page 12: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/12.jpg)
システムが“russia calls for joint front” まで出力した状態 次の出力単語 (against)をどう決めるか
途中状態で説明
![Page 13: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/13.jpg)
weighted average
次の単語 をどう決めるか 過去の自分の予測単語c個と入力文中の単語を利用
単語ベクトルの荷重平均ベクトル
×
![Page 14: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/14.jpg)
荷重はそのときの文脈情報でその都度決める
attention!!=
![Page 15: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/15.jpg)
式使った説明
![Page 16: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/16.jpg)
原文xが与えられた時の要約文yの条件付き確率
![Page 17: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/17.jpg)
今回の出力単語 過去c個の出力入力
![Page 18: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/18.jpg)
![Page 19: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/19.jpg)
calls for
ニューラル言語モデル[Bengio2003]文脈から次の単語を予測
softmax
大きく
![Page 20: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/20.jpg)
加重平均ベクトル を求める関数
3種類 うち一つが本命のattention
×
![Page 21: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/21.jpg)
エンコーダー1
単なる単語ベクトルの平均 - 過去の出力情報使わない - 全ての単語が同じ重み
使わない
![Page 22: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/22.jpg)
エンコーダー2
×… …
… …
- 畳み込み - max-pooling (size: 2) のセットをn回繰り返す
これも使わない
↑ は無いけどイメージとしては
![Page 23: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/23.jpg)
エンコーダー3
×
![Page 24: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/24.jpg)
エンコーダー3
×
![Page 25: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/25.jpg)
エンコーダー3
×…
…
![Page 26: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/26.jpg)
( )
エンコーダー3
×…
…
i
=
![Page 27: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/27.jpg)
( )
エンコーダー3
×…
…
i
=
![Page 28: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/28.jpg)
( )
エンコーダー3
×…
…
i
=
![Page 29: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/29.jpg)
エンコーダー3
×…
…
![Page 30: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/30.jpg)
エンコーダー3
×…
…
……
![Page 31: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/31.jpg)
エンコーダー3
×…
…weighted average
……
![Page 32: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/32.jpg)
エンコーダー3
×…
…weighted average
……
![Page 33: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/33.jpg)
×
![Page 34: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/34.jpg)
負の対数尤度を最小化
![Page 35: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/35.jpg)
ビームサーチ
![Page 36: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/36.jpg)
時には原文の単語をそのまま抽出した方が良いかもしれない
提案モデルunigram素性bigram素性trigram素性reordering素性
を学習することで 提案モデルのスコアが低い時はそのまま抽出
![Page 37: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/37.jpg)
細かい設定は割愛
DUC2003,2004の公式データ : 500事例 Gigaword corpusの一文目とタイトル : 400万事例
実験
![Page 38: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/38.jpg)
from authors slide http://people.seas.harvard.edu/~srush/emnlp2015_slides.pdf
抽出のやつ
![Page 39: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/39.jpg)
ROUGEによる既存研究との比較
![Page 40: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/40.jpg)
ROUGEは「正解との表層の被り」がスコアになるので Extraction要素を加えたABS+の方が良い
![Page 41: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/41.jpg)
場所や人などのキーワードは拾える !
構文的に誤った並べ替えが発生してしまったり
事例観察
![Page 42: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/42.jpg)
誤った主語
![Page 43: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/43.jpg)
人手要約者の「短くしたいバイアス」があるのでnzみたいな省略は 頻繁に起きている(はず)なので,対応が取れている(はず)
foreign minister→fmも同様
なんかすごい言い換え
+
![Page 44: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/44.jpg)
なんかすごい言い換えてるけど間違っている
more examples in the author’s slide: http://people.seas.harvard.edu/~srush/emnlp2015_slides.pdf
![Page 45: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/45.jpg)
![Page 46: A Neural Attention Model for Sentence Summarization [Rush+2015]](https://reader031.vdocuments.pub/reader031/viewer/2022030317/586e8b7e1a28aba0038b806f/html5/thumbnails/46.jpg)