preferred networksu Ì ô ¸ Ë è · preferred networksu Ì ô ¸ Ë è chainer, sensorbee...
TRANSCRIPT
![Page 1: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/1.jpg)
Preferred NetworksChainer, SensorBee
2016 7 23 ACM SIGMOD 62( 25 i i Web )
Preferred Networksg
[email protected] [email protected]
• (twitter: @delta2323_)– h → 2012.4 PFI → 2014.10 PFN– h g gChainer
• (twitter: @disktnk)– h → SIer → 2015.4 PFN– h eSensorBee e
2
![Page 2: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/2.jpg)
• Preferred Networks 4• Chainer
–– i i– Chainer t
• SensorBee– i i– i i SensorBee– Chainer SensorBee
3
Preferred Networks
• 2014 3 kPreferred Infrastructurex l• h e h ו h 40 ku 35 g i il•
– 2014 10 k lk2 l– 2015 8 FANUCk9 l– 2015 10 k10 l
• g– eFANUCe ePanasoniceNVIDIAeCiRA
4
![Page 3: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/3.jpg)
LML L L V R
• VD # y z v –i
•a y za t e z – z ™
•a – za t ™e ™
•a e i e i –
e ea – te x
-
AutomotiveHumanoid Robot
Preferred Networks Industrial IoT – AI
6
Consumer Industrial
Cloud
Device
PhotoGameText
Speech
Infrastructure
FactoryRobot
Automotive
Healthcare
Smart City
Industry4.0
Industrial IoT
![Page 4: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/4.jpg)
PFN
DIMo i
• 2 OSS te i1. SensorBee: x g i2. Chainer: x t t i i3. DIMo i : g – i
:IoT i i
: i i i i
Dee
p In
telli
genc
ein
-Mot
ion
(DIM
o)
…
???
…
7
• Preferred Networks– t• Chainer
–– i i– Chainer t
• SensorBee– i i– i i SensorBee– Chainer SensorBee
8
![Page 5: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/5.jpg)
=
• L y t y u–• ™
• L i t• w i y i
argminw∑(x, y) L(x, y; w)wh ixhyhLh
h k x – y l
9
Neural Network as a Computational Graph
• i i i i i i x DAGkDirected Acyclic Graphl z
y = x1 * x2z = y - x3
x1 mul suby
x3
z
x2
data node
operator node10
![Page 6: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/6.jpg)
kForward Propagationl
– i
ve –
x
w
_ ** 2
2 * _ _ * _
SCE L
_ + _ z
t
11
x = chainer.Variable(np.array(…))
# same as w, t
z = x ** 2 + 2 * x * w
L = SoftmaxCrossEntropy(z, t)
i k l
• 2• k L i w– e�wLl• w – x
initialize wuntil convergence:
get data (x, y)w← w - η�w L(x, y; w)
ηh
← – i ys
i – – z
12
![Page 7: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/7.jpg)
kChain Rulel
i
x f y
x f y
6 7
Forward Propagation
L e
13
kBackward Propagationl
• x ik l
x
w
_ ** 2
2 * _ _ * _
SCE L
_ + _ z
�z L
�w L
�x L
�L L = 1
t
�t L
14
![Page 8: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/8.jpg)
1. = + i→ i i – t
2.→ t –
3. –e i→ t –
15
L(x, y, w)
�w L
w← w - η�w L
• Preferred Networks– t• Chainer
–– d– Chainer t
• SensorBee– i i– i i SensorBee– Chainer SensorBee
16
![Page 9: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/9.jpg)
i i
17
Prepare the training dataset
Repeat until meeting some criterionPrepare for the next (mini) batch
Compute the loss (forward prop)
Initialize the NN parameters
Save the NN parameters
Define how to compute the loss of this batch
Compute the gradient (backprop)Update the NN parameters
automated
i i
name functions example
E D BG E (- - BFCE C E
B EB B CE CA B A BG
( G F G B A BGE B B /CCD
. E F / F BC F / EB
CADHG G CB E DA B A BG
H CADHG G CB E DCE E DECD DECD
BC BFCE CCE BB
H G A BF CBEE L E EL
/ B E E HA L H L) B GCE CE
HA E CADHG G CBD
GE K CD E G CBCBIC HG CB
/ H / H(
, E E
18
![Page 10: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/10.jpg)
i i
• NN i• NN i – 2 k l
– t i → i e y– i t g t → –
• t – i y– t i i es t y tNN
Deep Networks with Stochastic DepthG. Huang et al. 2016
Ladder Network A. Rasmus et. al. 2015
End-to-End Memory NetworkS. Sukhbaatar et. al. 2015
– k ly“ ” ” ” –
19
Chainer
• h 9 e 9
• h*()- . 1
• i
• a
.
• h= D 8
• NN
ü NN Pythonü i u NN
ü NN kDefine-by-Runlü Python i
ü i – NNü CuPy– CPU/GPU agnostic i
20
![Page 11: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/11.jpg)
cuDNN
Chainer
NumPy CuPy
BLAS cuBLAScuRAND
CPU GPU
name
E D BG E
B EB B CE CA B A BGCADHG G CB E D
A B A BGH G A BF CB
EE L E ELHA E CADHG G CB
D
, E E
* NumPyhPython f yw e Python i i yNumPy i – t
21
i i
Python C++ Lua Python C++/Python
n Preferred Networks /Infrastructure
BVLC Idiap Research Institute, DeepMind
Univ. of Montreal
2( n) o n n pjn
RNN/LSTM 1 c n n n
0 c c c c
0 p n n n
DSL(Python) (prototxt)
DSL(YAML)
DSLkPythonl
LuaJIT–i
i
× GPUg×i i
TensorBoard–
22
![Page 12: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/12.jpg)
f g
x f g
i i
x yf
x = chainer.Variable(...)y = f(x)z = g(x)
zg
i i3
7LMPUL B U 7LMPUL B U
v• Define-and-Run = NN• Define-by-Run = NN i23
i
yfor xi, yi in data:
x = Variable(xi)y = Variable(yi)z = x + 2 * y
7LMPUL B U7LMPUL B U
x = Variable(‘x’)y = Variable(‘y’)z = x + 2 * y
for xi, yi in data:eval(z, x=xi, y=yi))
24
![Page 13: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/13.jpg)
h i –MNIST
class MLP(Link):def __int__(self):super(MLP, self).__init__(
l1=Linear(784, 1000),l2=Linear(1000, 1000),l3=Linear(1000, 10))
def __call__(x):h1 = F.relu(self.l1(x))h2 = F.relu(self.l2(l1))return self.l3(h2)
model = L.Classifier(MLP()) optimizer = Adam()optimizer.setup(model)
Linear l1r
W bias
ReLU
Linear l2h1
W bias
ReLU
Linear l3h2
W bias
25
h i – MNISTtrain, test = get_mnist()train_iter = ShuffledIterator(train, 128)test_iter = SequentialIterator(test, 128, False)
updater = StandardUpdater(train_iter, optimizer)trainer = Trainer(updater, (10, 'epoch'))
trainer.extend(Evaluator(test_iter, model))trainer.extend(dump_graph('main/loss'))trainer.extend(snapshot())trainer.extend(LogReport())trainer.extend(PrintReport(
'epoch', 'main/accuracy','validation/main/accuracy']))
trainer.extend(ProgressBar())
trainer.run()
26
![Page 14: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/14.jpg)
Debuggubility
class MLP(Link):def __int__(self):super(MLP, self).__init__(
l1=Linear(700, 1000), # Oops!l2=Linear(1000, 1000),l3=Linear(1000, 10))
27
Chainer –w
• t – ze y t• NVIDIAy i i i• Python NumPy i i y x v• th e e ie• (PFN ) ve i y t
NVIDIA CEONVIDIA cuDNN 28
![Page 15: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/15.jpg)
• Preferred Networks– t• Chainer
–– i i– Chainer 4
• SensorBee– i i– i i SensorBee– Chainer SensorBee
29
t i
30
CES 2016– eNTT
![Page 16: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/16.jpg)
t –
(iREX) i –
31
–
t
x i
(iREX) i –
32
![Page 17: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/17.jpg)
–
b
)-
(iREX) i –
33
– ™
Panasonic
34
![Page 18: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/18.jpg)
–
http://qiita.com/advent-calendar/2015/chainer#day-13
35
• Preferred Networks– t• Chainer
–– Chainer– Chainer t
• SensorBee–– i i SensorBee– Chainer SensorBee
36
![Page 19: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/19.jpg)
l i y u–a ) )((D5e i e
IoT
37
l i –a i e ia i 2 i i tg z ta 2 i – i t
IoT i
38
![Page 20: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/20.jpg)
l i x – t ta i i ia y – e u
i i
39
i i
• – i– – i y– 7 x 7
• i– i i e e e i etc– i e u
- m m- m × m i
- x→
• x i – y i→ b
40
![Page 21: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/21.jpg)
• Preferred Networks– t• Chainer
–– Chainer– Chainer t
• SensorBee– i i– SensorBee– Chainer SensorBee
41
i – t
• – ™ e i–
- 1- – (= i) z ( )
–- (Continuous Queries)
- t y- tx– – x
– 1 event (=tuple) t
• i i i– Strom, Spark Streaming, Samza, Kafka Streaming, Flink, Heron...
( e y t t)
42
![Page 22: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/22.jpg)
i
• Data Stream Management System (DSMS)
• C L T 7 S MV T
6VU PU V A L PLC L TPUN U C L TPUN
L T
)
+
,*
http://www.confluent.io/kafka-summit-2016-keynote-the-rise-of-real-time-jay-kreps p10
R :VS I = D TL Özsu. Data Stream Management, 2010.
43
SensorBee: Intelligent Streaming ETL for IoT
• i s t i (Extract) w(Transform) v i
• (*1) x t fGo f• BQL tu SQL – i (*2)
6 TL
FLOP SL
BV L
CLU V
MS LU
B75=C
CL O8UNPUL
VCA
7 CV L 7 7L PU PVU
CLU V 5LL
) O 2 P V VT T LUG VS PVU LU PV V VT PUN V L PL M * O 2 SP L O L UL P R UR LU V ILL -10/(0// ),
44
![Page 23: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/23.jpg)
SensorBee: Flexibility and Extensibility
• jubatus ( g)
• opencv epymlstateetc...
6 TL
FLOP SL
BV L
CLU V
MS LU
B75=C
CL O8UNPUL
VCA
7 CV L 7 7L PU PVUCLU V 5LLd
S N PU S N PU
E79
E L 7LMPUL 9 U PVU E79 E L7LMPUL C L E7C –
e y
E7C
= OPUL SL UPUNb 6S PMP PVUb BLN L PVUb IQL L VNUP PVUb 4UVT S_ L L PVU
45
• Preferred Networks– t• Chainer
–– Chainer– Chainer t
• SensorBee– i i– i i SensorBee– Chainer SensorBee y
46
![Page 24: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/24.jpg)
1: Twitter • z
•
a D V P SO 2 LU V ILL L OL V V N LU S L V P S O TS PUNT OPUL SL UPUN
a Elasticsearch –http://www.slideshare.net/nobu_k/elasticsearch-59627321
D P L
:LUL
4NL
9V TPUN ILSP
UN MS LU
47
6
CLU V 5LL
2: t i
4B×i i
4B×i i
DD i
DD i
6
CLU V 5LL
48
![Page 25: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/25.jpg)
3:
SensorBee
6VT L P PVU7L L D R BL VNUP L
http://monoist.atmarkit.co.jp/mn/articles/1512/07/news083.html ( i )http://techon.nikkeibp.co.jp/atcl/news/16/041801679/?i_cid=nbptec_sied_rel ( i )http://www.fanuc.co.jp/ja/whatsnew/notice/osirase20160418.html ( i )49
4: –
SensorBee
DPTL L PLB BL L LU PVU
50
![Page 26: Preferred Networksu Ì ô ¸ Ë è · Preferred Networksu Ì ô ¸ Ë è Chainer, SensorBee 2016Ï7á23ÍACM SIGMODÍÒ E÷ 962ú E÷ÎÑ ( 925úii Web ² 7 óÑ) q ÑÝPreferred](https://reader035.vdocuments.pub/reader035/viewer/2022081502/5fb68e1b51271e5ac50619ec/html5/thumbnails/26.jpg)
• Chainer define-by-run i iflexibility debuggabilityy
• sensorbee UDFgUDS– IoT i
• PFN Chainer– SensorBee– –eR&D t
51
52