「知識」のdeep learning
TRANSCRIPT
Deep Learning
1 Preferred Infrastructure (@unnonouno)
2015/06/04 PFI
! 1 !
!
NLP 12014-
NLP 1YANS
! YANS 19
! 140
! ! :l
! YANS 13 ! ! "
Knowledge Representation
1.
3.
4.
2.
1.
3.
4.
2.
1.
3.
4.
2.
RNN
Recurrent Neural Network Language Model (RNNLM) [Mikolov+10]
! RNN ! !
! LSTM�T���������� ��
Long Short-Term Memory (LSTM)
!
!
1 ( 05$!
RNN1LSTM
! !
! V�;f�
! b�K����1+3
argmaxy P(y|x) = argmaxy P(x|y) P(y)
1.
3.
4.
2.
[ 15]
!
:A@�A[�1���� ! -‐‑‒
-5.4&,
-‐‑‒ [ 15]
! Pydata Tokyo
pS�Qg�D]
1.
3.
4.
2.
-‐‑‒
!
!
!
Knowledge Representation
! EMNLP2014 Bordes Weston1 part 2 [Bordes&Weston14]
! [15]
Q: Deep LearningRepresentation Learning
-‐‑‒ A: Deep
推論を導けるような知識の表現、およびその方法を開発する人工知能研究の領域である。
! ! ! 1
1
! 2 t t 3
! t t s 2/3X�u
! RDF
(x, r, y)
x yr
1Knowledge Base
! New York NY
! E�L1+3/#,3�qO
! t t ! t t ! t t
-‐‑‒
!
-‐‑‒
NLP "-4)%�Y<����
! {(xi, ri, yi)}:
! x, y: t t
! r: t t
! x, y ! r
! !
1
argmax ∑i f(xi, ri, yi)
Distance model (Structured Embedding) [Bordes+11]
! e ! Rleft, Rright
!
! f
f(x, r, y) = || Rleft(r) e(x) – Rright(r) e(y) ||1
TransE model [Brodes+13]
! r /#,3 r �]��
!
f(x, r, y) = || e(x) + r – e(y) ||22
TransE model
x
y
r
TransE model
1 TransEB��r�N�/#,3�?8
ia ! TransM: ! TransH:
TransM model [Fan+14]
! r ! wr r x, y
f(x, r, y) = wr|| e(x) + r – e(y) ||22
TransH model [Wang+14]
! dJR6�mc TransE
Bilinear model
! r !
f(x, r, y) = e(x)T Wr e(y)
Neural Tensor Network (NTN) [Socher+13]
f(x, r, y) = ur tanh(e(x)Wre(y) + V1
r e(x) + V2r e(y) + br)
! r *5'3 !
[Yang+15]
! 2
! Bilinear
Q: -‐‑‒
A: 9n<P [Nickel+11]
Link prediction
! t t 1 ! QA
! -‐‑‒
! t t
(e1, r, e2) (e1, r, ? )
TransH
TransE
[Bordes&Weston14]
! !
[Weston+13]
! x y VZ: r !
[Bordes&Weston14]
[Weston+13]
! !
-‐‑‒
[Bordes&Weston14]
[Bordes&Weston14]
Link prediction 1QA
! Link prediction
!
QA [Bordes+14]
!
_FV�>H ! q t
f(q) g(t)
! f(q), g(t) q, t
Memory networks: [Weston+15]
! I: I(x)
! G: mi = G(mi, I(x), m) ! O: o = O(I(x), m) ! R: r = R(o)
I:
O:
R:
G:
!
! !
!
_F\h1+3
! Deep Learning -‐‑‒
-‐‑‒
! ! -‐‑‒ ! -‐‑‒
! -‐‑‒ ! ! factoid
! -‐‑‒ !
RNN [Peng&Yao15]
! RNN
!
! ! !
! !
!
! !
1/4
! [Mikolov+10] T. Mikolov, M. Karafiat, L. Burget, J. H. Cernocky, S. Khudanpur. Recurrent neural network based language model. Interspeech 2010.
! [ 15] . kUCIv7���!2�<t.
2015. ! [ 15] .
NLP Introduction based on Project Next NLP. PyData.Tokyo Meetup #5, 2015.
! [Bordes&Weston14] A. Bordes, J. Weston. Embedding Methods for Natural Language Processing. EMNLP2014 tutorial.
! [ 15] . j`�<oCI�^HG�MW1+3�=e. JSAI 2015 .
2/4
! [Bordes+11] A. Bordes, J. Weston, R. Collobert, Y. Bengio. Learning structured embeddings of knowledge bases. AAAI2011.
! [Bordes+13] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, O. Yakhnenko. Translating Embeddings for Modeling Multi-relational Data. NIPS 2013.
! [Fan+14] M. Fan, Q. Shou, E. Chang, T. F. Zheng. Transition-based Knowledge Graph Embedding with Relational Mapping Properties. PACLIC 2014.
! [Wang+14] Z. Wang, J. Zhang, J. Feng, Z. Chen. Knowledge Graph Embedding by Translating on Hyperplanes. AAAI 2014.
3/4
! [Socher+13] R. Socher, D. Chen, C. D. Manning, A. Y. Ng. Reasoning With Neural Tensor Networks for Knowledge Base Completion. NIPS 2013.
! [Yang+15] B. Yang, W. Yih, X. He, J. Gao, L. Deng. Embedding Entities and Relations for Learning and Inferenece in Knowledge Bases. ICLR 2015.
! [Nickel+11] M. Nickel, V. Tresp, H. P. Kriegel. A Three-Way Model for Collective Learning on Multi-Relational Data. ICML 2011.
4/4
! [Bordes+14] A. Bordes, J. Weston, N. Usunier. Open Question Answering with Weakly Supervised Embedding Models. ECML/PKDD 2014.
! [Weston+15] J. Weston, S. Chopra, A. Bordes. Memory Networks. ICLR 2015.
! [Peng&Yao15] B. Peng, K. Yao.
Recurrent Neural Networks with External Memory for Language Understanding. arXiv:1506.00195, 2015.