1. 1. 2017.05.18 D1 Masayoshi Kondo - About Reading Comprehension@2017 Linguistic Knowledge as Memory for Recurrent Neural Networks arXiv:05/07 ver.1 Bhuwan Dhingra, Zhilin Yang, William W.Choen, Ruslan Salakhutdinov
2. 2. XX: . . .
3. 3. --: Whats Reading Comprehension Task ? / Reading Comprehension Task () Sqatriplet:(S, q, a) (S, q, ?)[?] (Sa or qa11) [bAbi dataset] (entity) (entity) [LAMABADA dataset] ()
4. 4. --: Reading Comprehension bAbi(and bAbi-mix) / LAMBADA / CNN (accuracy-) abstract RNN () DAGs RNN(memory) text comprehensionCNN, bAbi, LAMBADA (achieve new state-of-the-art)bAbi QA 1000example2015 representation()
5. 5. 1. Introduction 2. Related Work 3. Methods 4. Experiments 5. Results
6. 6. . Deep Learning RNN 01: Introduction CellLSTM(CEC), GRU Attention Mechanism Memory Networks Daniluk et al.(2017) showed that even memory-augmented neural models do not look beyond the immediately preceding time steps. Frustratingly short a/ention spans in neural language modeling [arXiv:1702.04521, 2017]
7. 7. /(given) 02: Introduction : sequential links / : coreference relations / : hypohypernymy Mary got the football. She went to the kitchen. She left the ball there. (extract relations: coreference, hypernymy) sequential links the content of memory: linkrepresentationedge labeledge label RNN Sequential links Memory Graph (relation) / link representation/edge label MAGE RNN
8. 8. 03: Introduction - MAGE RNN (the Memory as Acyclic Graph Encoding RNN/GRU) () DAGs(Directed Acycle Graph) DAG MAGE-RNNtext comprehension taskcoreference relation text comprehension RNN MAGE-GRUGRU CNN dataset / bAbi QA tasks / LAMBADA dataset representationentity
9. 9. 1. Introduction 2. Related Work 3. Methods 4. Experiments 5. Results
10. 10. 04: Related Work Graph Neural Networks - The graph neural network model [Scarselli et al, IEEE09] Gated Graph Sequential Neural Networks - Gated graph sequential neural netoworks [Li et al, ICLR016] : () (). / O(node-size*time-step) Reading comprehension : - Emergent logical structure in vector representations of neural readers [Wang et al, Preprint2017] Tracking the world state with recurrent entity networks. [Hena et al, xrXiv2017] - Recurrent Entity Network architecture. -coreference representationentity.
11. 11. 1. Introduction 2. Related Work 3. Methods 4. Experiments 5. Results
12. 12. -1: 1. (coreference) 2. sequence links coreference 3. 4. DAG 5. sequence links 6. -(1) triplet:nodenode typeAtriplet-(node, node, edge-typeA) 7. -(2)type triplet 8. -(3)type concat 9. -(4) 7
13. 13. -2: -1 : sequential links / : coreference relations / : hypohypernymy Mary got the football. She went to the kitchen. She left the ball there. () Stanford CoreNLP Parser type 1. (coreference) 2. sequence links coreference 3.
14. 14. -3: -1 4. DAG 5. sequence links
15. 15. -4: -1 6. -(1) triplet : node node typeA > triplet-(node, node, edge-typeA) mary football she went she () she triplet = { (mary, she, red), (football, she, black) } f (she) f (she)
16. 16. ht e ~ act _ func W e xt + Ue,e' hk e' (k,e') + bh e $ % && ' ( )) -5: -1 7.-(2)type triplet t node t hk orange hk blue hk black k node k type type torange-type()k type(sum)
17. 17. -6: -1 8. -(3)type concat t node t type orange black blue green t type ht orange ht black ht blue ht green concatenate ht ()RNN(GRU) 9. -(4)
18. 18. 05: Methods inner-edgetriplet type(GRUcell)
19. 19. 06: Methods typeconcatenate .(GRUcell) if (t',ei ) (xt ):gt ei = ht' ei else : gt ei = 0 gt = gt e1 || gt e2 || gt e3 ||... || gt eE ttype:ei t t type:ei concat.
20. 20. 1. Introduction 2. Related Work 3. Methods 4. Experiments 5. Results
21. 21. 07: Experiments : Story Based QA bAbi dataset bAbi dataset : bAbi-mix dataset : bAbi dataset 20toy task (entity) chaining facts, counting, deduction, induction reasoning ability bAbi-mix: 1. 2. Entity mention 3. ( 4. DAG
22. 22. 08: Experiments : Story Based QA bAbi dataset Error rate on the 20 bAbi tasks. Task 3 : [Class name] WhereWasObject / Factoid QA with three supporting facts Task 16 : [Class name] Induction / Basic induction () GA : GA Reader - Gated-attention readers for text comprehension [arXiv:1606.01549, 2016] RQN : Query Reduction Networks - Query-reduction networks for question answering [Seo et al, iclr17]
23. 23. 09: Experiments : Story Based QA bAbi dataset Error rate over 20tasks on bAbi mix. QRNreading comprehension error rateTask 3 error rate
24. 24. 10: Experiments : Story Based QA bAbi dataset Task 7 () Sandra Sandra None Sandra 1 Daniel 1 Sandra 2 ()
25. 25. 11: Experiments : Story Based QA bAbi dataset Task 8 () () John Daniel Nothing Jhon Apple Sandora Apple Jhon Nothing Daniel Nothing Jhon Apple
26. 26. 12: Experiments : Broad Context Language ModelingLAMBADA dataset 4,5 sentence LM7.3% Chu et al(2016) 49% Chu Stanford CoreNLP Parser ( extract co-reference chains)
27. 27. 13: Experiments : Broad Context Language Modeling Table 4GAco-referenceGA+MARE Chu et al(2016)100 validsingle name cue, semantic trigger, coreference, external knowledge labels.
28. 28. 14: Experiments : Broad Context Language Modeling GA GA+MAGE entity
29. 29. 15: Experiments : Cloze-style QACNN dataset Accuracies on CNN dataset named entity (Stanford CoreNLP Parser) (entity-id 0.7%
30. 30. 1. Introduction 2. Related Work 3. Methods 4. Experiments 5. Results
31. 31. 16: Results RNN MAGE-RNNtype ()type coreferenceMachine comprehension model dependency parsing, semantic role label, semantic frames, ontologies(Wordnet), database(Freebase) typeattention MAGE-RNN attention
32. 32. END