semi-supervised classification with graph convolutional networks @iclr2017読み会

Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

June 17, 2017 ICLR @DeNA Eiji Sekiya AI Research and Development Gr. AI System Dept. DeNA Co., Ltd.

1

h"ps://openreview.net/pdf?id=SJU4ayYgl

https://openreview.net/pdf?id=SJU4ayYgl


•

• (@eratostennis)

•

• Master

• DeNA (2014)

• : (~2016)

• Hadoop, Vertica

• : (2016~)

•

• GameAI2


Abstract

• Semi-supervised learning on graph-structured data

• Localized first-order approximation of spectral graph• :

•

•

3


Agenda

• Introduction

• Related Work

• Fast Approximate Convolutions on Graphs

• Semi-Supervised Node Classification

• Experiments

• Results

• Conclusion

4


Relation to Weisfeiler-Lehman Algorithm

• WL-1

• Hash Neural Network

5

hash


Node Embeddings with Random Weights

6

output 2


Semi-Supervised Node Embeddings

• 4 1

7:h"p://tkipf.github.io/graph-convoluDonal-networks/


Related Work

• Graph-Based Semi-Supervised Learning

• Graph Laplacian regularization

• Label Propagation (Zhu et al., 2003)

• Manifold Regularization (Belkin et al., 2006)

• Graph embedding-based

• DeepWalk (Perozzi et al., 2014)

• Planetoid (Yang et al., 2016)

8


Related Work (DeepWalk: Perozzi et al., 2014)•

• SkipGram embedding

• Random Walk

9


Related Work

• Graph-Based Semi-Supervised Learning

• Graph Laplacian regularization

• Label Propagation (Zhu et al., 2003)

• Manifold Regularization (Belkin et al., 2006)

• Graph embedding-based

• DeepWalk (Perozzi et al., 2014)

• Planetoid (Yang et al., 2016)

• Neural Networks on Graphs ( )

• Spectral graph convolutional neural network (Bruna et al., 2014)

• Fast localized convolution (Defferrard et al., 2016)

10


Graph-Based Semi-Supervised Learning

• (citation network ) (documents)

•

•

•

• encode

• 11

Neural Networks on Graph


Multi-layer Graph Convolutional Network (GCN)

•

•

•

•

12

Contribution


Spectral Graph Convolutions

• Convolution with filter

• :

•

• …(Hammond at el., 2011)

•

• ( )

13


Spectral Graph Convolutions (Defferrad at el., 2016)

• Graph Convolution ( , )

• K K

• :

•

14


Layer-Wise Linear Model (Contribution)

• K=1 …

• k k

•

• Renormalization Trick ( )

• Input C, F …

•

15


Semi-Supervised Node Classification

• Forward Model ( )

• Cross-Entropy:

16


Experiments (Dataset)

• (Citeseer, Cora, Pubmed)

• Sparse bag-of-words feature vectors for each document

• a list of citation links between documents

• (NELL)

• A set of entities connected with directed, labeled edges

• Random Graph

• A dataset with N nodes we create a random graph assigning 2N edges uniformly at random

17


Experiments (Setup)•

• 2 GCN

•

• Prediction Accuracy

• 1000

•

• 500

• Dropout rate, L2 regularization factor, Number of hidden units

• Cora Citeseer Pubmed

18


Experiments (Baseline)

• Baseline

• Label Propagation (LP)

• Semi-supervised embedding (SemiEmb)

• Manifold regularization (ManiReg)

• Skip-gram based graph embeddings (DeepWalk)

• Iterative classification algorithm (ICA)

• Planetoid

19


Results (Semi-Supervised Node Classification)

• Results for all other baseline methods are taken from the Planetoid paper

• Environment

• The same hardware as planetoid

20

TrainingDmeinseconds

unDlconvergence

• Compare different variants of our proposed per-layer propagation model on the citation network datasets


Results (Evaluation Propagation Model)

21


Results (Training Time per Epoch)

• The mean training time per epoch for 100 epochs on simulated random graphs

22


Experiments on Model Depth

• Residual connections

• 2, 3

• 7 …

• residual connections

•

23


Discussion

• Keypoints

• Overcome both limitations

• Graph-Laplacian regularization

• Edges encode mere similarity of nodes

• Skip-gram based methods

• Limited by the fact that they are based on a multi-step pipeline which difficult to optimize

• Findings

• Propagation from neighboring nodes improves classification performance

• Proposed renormalized propagation model offers both improved efficiency

24


Future Work

• Memory Requirement

• Full-batch gradient descent

• Grows linearly in the size of the dataset

• Mini-batch stochastic gradient descent can alleviate this issue

• Directed edges and edge features

• Limited to undirected graph

• NELL show it is possible to handle

• By representing the original directed graph as an undirected bipartite graph

• Limiting assumptions

• Locality

• dependence on the Kth-order neighborhood for a GCN with K layers

• Equal importance of self-connections vs. edges to neighboring nodes 25


Reference• Graph Semi-Supervised Learning

• Xiaojin Zhu, Zoubin Ghahramani, and John Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In International Conference on Machine Learning (ICML), volume 3, pp. 912–919, 2003.

• Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of machine learning research (JMLR), 7(Nov):2399–2434, 2006.

• JasonWeston,Fre de ricRatle,HosseinMobahi,andRonanCollobert. Deeplearningviasemi- supervised embedding. In Neural Networks: Tricks of the Trade, pp. 639–655. Springer, 2012.

• Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in neural information processing systems (NIPS), pp. 3111–3119, 2013.

• Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701–710. ACM, 2014.

• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016.

26


Reference• Dataset & Pre-Processing

• Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29(3):93, 2008.

• Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam R. Hruschka Jr, and Tom M. Mitchell. Toward an architecture for never-ending language learning. In AAAI, volume 5, pp. 3, 2010.

• Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning (ICML), 2016.

• Website

• http://tkipf.github.io/graph-convolutional-networks/

• Implementation

• https://github.com/tkipf/gcn

27

http://tkipf.github.io/graph-convolutional-networks/

https://github.com/tkipf/gcn

semi-supervised classification with graph convolutional networks @iclr2017読み会

Data & Analytics