2016 dg2
TRANSCRIPT
10/06/20162/24
Notice
次回以降で取り上げる話題，データ，プロジェクト，質疑応答のために
slack のチームを作成しました。今回参加されなかった方でも自由に参
加できます。チーム名は deeppython.slack.com です。ご参加ください。
参加ご希望の方は [email protected] までメールをお
願いします。
＃DLｗPY
10/06/20163/24
本日のお品書き● Basic knowledge of neurons● Cross entropy ● Feedforwards and feedbacks of neural networks● Try googleplayground ● Convolutional neural networks● Try keras
10/06/20164/24
前回の補足最速で理解するには
1. 線形回帰 linear regression
2. ロジスティック回帰 logistic regression
3. 正則化 regularization
4. 多層パーセプトロン multi-layered perceptrons
5. 畳み込みニューラルネットワーク convolutional neural networks
6. リカレントニューラルネットワーク recurrent neural networks
7. 強化学習 reinforcement learning
10/06/20165/24
Warning: Stealing Machine Learning Models viaPrediction APIs https://arxiv.org/abs/1609.02943
Machine learning (ML) models may be deemed confidential due to their sensitive training data,commercial value, or use in security applications. Increasingly often, confidential ML models arebeing deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics")systems are an example: Some allow users to train models on potentially sensitive data and chargeothers for access on a pay-per-query basis.
The tension between model confidentiality and public access motivates our investigation of modelextraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge ofan ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") themodel. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partialfeature vectors as inputs and include confidence values with predictions. Given these practices, weshow simple, efficient attacks that extract target ML models with near-perfect fidelity for popularmodel classes including logistic regression, neural networks, and decision trees. We demonstratethese attacks against the online services of BigML and Amazon Machine Learning. We further showthat the natural countermeasure of omitting confidence values from model outputs still admitspotentially harmful model extraction attacks. Our results highlight the need for careful ML modeldeployment and new model extraction countermeasures.
10/06/20166/24
Prerequisites
● Basics about Neural Networks– How the brain actually works?
– How parallel computation works adapting parameters inspired byneurons?
– How the brain implements learning algorithms?
● (ペ)What is it a good idea to try to emulate the brain when solvinga recognition task?
10/06/20167/24
A schematic neuron
There are many neurotransimtters, butwe deal with those as positive/negativeweights and also positive negativeinputs. (ペ) why?
http://www.mhhe.com/socscience/intro/ibank/set1.htm
10/06/20169/24
Cross entropy
10/06/201611/24
Convnet.js http://cs.stanford.edu/people/karpathy/convnetjs/
10/06/201612/24
TensorFlow Tips
● Computation graph (see http://colah.github.io/posts/2015-08-Backprop/ , その翻訳記事は http://postd.cc/2015-08-backprop/ )
● Ways of installations (c.f. Tensorflow.org Download and Setup )– Pip
– Virutalenv
– Anaconda
– Docker
● Let’s try http://playground.tensorflow.org/● You can also check it out, Karpthy’s convnetjs● Keras is another choice to consider
10/06/201613/24
A computation graph
http://deeplearning.net/software/theano/extending/graphstructures.html
10/06/201614/24
Sample code of TensorFlow
import tensorflow as tfW = tf.get_variable(shape=[], name='W')b = tf.get_variable(shape=[], name='b')x = tf.placeholder(shape=[None], dtype=tf.float32, name='x')y = tf.matmul(W, x) + bwith tf.Session() as sess: sess.run(tf.initialize_all_variables()) print(sess.run(y, feed_dict={x: x_in}))
10/06/201615/24
The difference between placeholder and variableSince Tensor computations compose graphs then it's better to interpret the two in terms of graphs.
When you launch the graph, variables have to be explicitly initialized before you can run Ops that usetheir value. Then during the process of the an operation variables should be constant.
import tensorflow as tf
# Create a variable.# w = tf.Variable(<initial-value>, name=<optional-name>)w = tf.Variable(tf.truncated_normal([10, 40]))v = tf.Variable(tf.truncated_normal([40, 20]))
# Use the variable in the graph like any Tensor.# The variable should be initialized before this operation!y = tf.matmul(w, v)
# Assign a new value to the variable with `assign()` or a related method.w.assign(w + 1.0)w.assign_add(1.0)
http://stackoverflow.com/questions/36693740/whats-the-difference-between-tf-placeholder-and-tf-variable
tf.Variableはオペレーション実行前に初期化される
10/06/201616/24
The difference between placeholder and variableA placeholder is a handle of a value in the operation and it can be not initialized before the execution ofthe graph (launching the graph in the session which does its computation relaying on a highly efficient C++ backend).
x = tf.placeholder(tf.float32, shape=(1024, 1024))# You don't need to initialize it to calculate y, it's different from # the variable above, the placeholder is a "variable"(not intialized) # in this operation. y = tf.matmul(x, x)
with tf.Session() as sess: # However you should initialize x to execute y for the execution of the graph. print(sess.run(y)) # ERROR: will fail because x was not fed.
rand_array = np.random.rand(1024, 1024) print(sess.run(y, feed_dict={x: rand_array})) # Will succeed.
プレースホルダーは初期化されない
10/06/201617/24
Convolutional neural networks: LeNet5
LeCun1998 より
10/06/201618/24
AlexNet
Krizensky+2012 より
10/06/201619/24
GoogLeNet
10/06/201620/24
GoogLeNet Inception module
10/06/201621/24
畳み込み演算
10/06/201622/24
kernels
10/06/201623/24
To understand convolution
● 畳み込みの理解には http://colah.github.io/posts/2014-07-Understanding-Convolutions/
10/06/201624/24
What is convolution?
● Convolution is an operation from signal processing● Filters, or kernels in machine learning,