semi-supervised classification with graph convolutional networks @iclr2017読み会

June 17, 2017 ICLR @DeNA Eiji Sekiya AI Research and Development Gr. AI System Dept. DeNA Co., Ltd.

h"ps://openreview.net/pdf?id=SJU4ayYgl

• (@eratostennis)

• Master

• DeNA (2014)

• : (~2016)

• Hadoop, Vertica

• : (2016~)

• GameAI2

Abstract

• Semi-supervised learning on graph-structured data

• Localized first-order approximation of spectral graph• :

Agenda

• Introduction

• Related Work

• Fast Approximate Convolutions on Graphs

• Semi-Supervised Node Classification

• Experiments

• Results

• Conclusion

Relation to Weisfeiler-Lehman Algorithm

• WL-1

• Hash Neural Network

Node Embeddings with Random Weights

output 2

Semi-Supervised Node Embeddings

• 4 1

7:h"p://tkipf.github.io/graph-convoluDonal-networks/

Related Work

• Graph-Based Semi-Supervised Learning

• Graph Laplacian regularization

• Label Propagation (Zhu et al., 2003)

• Manifold Regularization (Belkin et al., 2006)

• Graph embedding-based

• DeepWalk (Perozzi et al., 2014)

• Planetoid (Yang et al., 2016)

Related Work (DeepWalk: Perozzi et al., 2014)•

• SkipGram embedding

• Random Walk

Related Work

• Graph-Based Semi-Supervised Learning

• Graph Laplacian regularization

• Label Propagation (Zhu et al., 2003)

• Manifold Regularization (Belkin et al., 2006)

• Graph embedding-based

• DeepWalk (Perozzi et al., 2014)

• Planetoid (Yang et al., 2016)

• Neural Networks on Graphs ( )

• Spectral graph convolutional neural network (Bruna et al., 2014)

• Fast localized convolution (Defferrard et al., 2016)

Graph-Based Semi-Supervised Learning

• (citation network ) (documents)

• encode

• 11

Neural Networks on Graph

Multi-layer Graph Convolutional Network (GCN)

Contribution

Spectral Graph Convolutions

• Convolution with filter

• …(Hammond at el., 2011)

• ( )

Spectral Graph Convolutions (Defferrad at el., 2016)

• Graph Convolution ( , )

• K K

Layer-Wise Linear Model (Contribution)

• K=1 …

• k k

• Renormalization Trick ( )

• Input C, F …

Semi-Supervised Node Classification

• Forward Model ( )

• Cross-Entropy:

Experiments (Dataset)

• (Citeseer, Cora, Pubmed)

• Sparse bag-of-words feature vectors for each document

• a list of citation links between documents

• (NELL)

• A set of entities connected with directed, labeled edges

• Random Graph

• A dataset with N nodes we create a random graph assigning 2N edges uniformly at random

Experiments (Setup)•

• 2 GCN

• Prediction Accuracy

• 1000

• 500

• Dropout rate, L2 regularization factor, Number of hidden units

• Cora Citeseer Pubmed

Experiments (Baseline)

• Baseline

• Label Propagation (LP)

• Semi-supervised embedding (SemiEmb)

• Manifold regularization (ManiReg)

• Skip-gram based graph embeddings (DeepWalk)

• Iterative classification algorithm (ICA)

• Planetoid

Results (Semi-Supervised Node Classification)

• Results for all other baseline methods are taken from the Planetoid paper

• Environment

• The same hardware as planetoid

TrainingDmeinseconds

unDlconvergence

• Compare different variants of our proposed per-layer propagation model on the citation network datasets

Results (Evaluation Propagation Model)

Results (Training Time per Epoch)

• The mean training time per epoch for 100 epochs on simulated random graphs

Experiments on Model Depth

• Residual connections

• 2, 3

• 7 …

• residual connections

Discussion

• Keypoints

• Overcome both limitations

• Graph-Laplacian regularization

• Edges encode mere similarity of nodes

• Skip-gram based methods

• Limited by the fact that they are based on a multi-step pipeline which difficult to optimize

• Findings

• Propagation from neighboring nodes improves classification performance

• Proposed renormalized propagation model offers both improved efficiency

Future Work

• Memory Requirement

• Full-batch gradient descent

• Grows linearly in the size of the dataset

• Mini-batch stochastic gradient descent can alleviate this issue

• Directed edges and edge features

• Limited to undirected graph

• NELL show it is possible to handle

• By representing the original directed graph as an undirected bipartite graph

• Limiting assumptions

• Locality

• dependence on the Kth-order neighborhood for a GCN with K layers

• Equal importance of self-connections vs. edges to neighboring nodes 25

Reference• Graph Semi-Supervised Learning

• Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In International Conference on Machine Learning (ICML), volume 3, pp. 912–919, 2003.

• Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of machine learning research (JMLR), 7(Nov):2399–2434, 2006.

• JasonWeston,Fre de ricRatle,HosseinMobahi,andRonanCollobert. Deeplearningviasemi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639–655. Springer, 2012.

• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems (NIPS), pp. 3111–3119, 2013.

• Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701–710. ACM, 2014.

• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016.

Reference• Dataset & Pre-Processing

• Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93, 2008.

• Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr, and Tom M. Mitchell. Toward an architecture for never-ending language learning. In AAAI, volume 5, pp. 3, 2010.

• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016.

• Website

• http://tkipf.github.io/graph-convolutional-networks/

• Implementation

• https://github.com/tkipf/gcn

semi-supervised classification with graph convolutional networks @iclr2017読み会

Data & Analytics

convolutional neural network - seoul national...

convolutional neural networks for sentiment classification

convolutional neural networks

autentikasi daun herbal menggunakan convolutional neural

densely connected convolutional networks

implementasi metode convolutional neural network …

implementasi convolutional neural networks untuk

understanding convolutional neural networks with

algorithmic graph theory part ii - graph...

convolutional codes

implemetasi algoritma convolutional neural …

klasifikasi citra x-ray menggunakan convolutional …

convolutional neural network - unnes

supervised injecting facilities ‘the case’

convolutional neural network untuk identifikasi …

« convolutional deep belief networks »

graph homomorphism revisited for graph matching

implementasi algoritma convolutional recurrent …

convolutional neural networks cnn partie i

perbandingan metode klasifikasi supervised...