同济大学软件学院
深度学习基础
张林同济大学2018年12月
同济大学软件学院
10 Breakthrough Technologies 2017 (MIT Tech Review)
Reversing Paralysis Self Driving Paying with Your Face
Practical Quantum Computers
The 360-Degree Selfie
Hot Solar Cells Gene Therapy 2.0 The Cell Atlas Botnets of Things Reinforcement Learning
The core is machine learning
同济大学软件学院
傍晚,小街路面上沁出微雨后的湿润,和煦的细风吹来,抬头看看天边的晚霞,嗯,明天又是一个好天气。走到水果摊旁,挑了个根蒂蜷缩、敲起来声音浊响的青绿西瓜,一边满心期待着皮薄肉厚瓤甜的爽落感,一边愉快地想着:这学期狠下了功夫,基础概念弄得清清楚楚,算法作业也是信手拈来,这门课成绩一定差不了!
摘自《机器学习》(周志华著,2016)
同济大学软件学院
Outline
• Basic concepts• Linear model• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• DCNN for object detection
同济大学软件学院
A little history about AI
1956年,麦卡锡召集哈佛大学、麻省理工学院、IBM公司、贝
尔实验室的研究人员召开达特茅斯会议正式提出“人工智能”
人工智能是指计算机系统具备的能
力,该能力可以履行原本只有依靠人
类智慧才能完成的复杂任务
人工智能1
2006年达特茅斯会议当事人重聚,左起:摩尔、麦卡锡、明斯基、
赛弗里奇、所罗门诺夫
John McCarthy
人工智能之父
同济大学软件学院
A little history about AI
人工智能的最终目标
人工智能
探讨智能形成的基本机理,研
究利用自动机模拟人的思维过程
人工智能的近期目标
研究如何使计算机去做那些靠
人的智力才能做的工作
同济大学软件学院
A little history about AI
人工智能的研究范式及历程1
符号主义:采用知识表达和逻辑符号系统来模拟人类的智能,试图对智能进行宏观研究(Knowledge-driven)
联接主义:始于W.S. McCulloch和皮兹(Pitts)的先驱工作,直到目前的深度学习,是微观意义上的探索(Data driven)
生物启发的智能:依赖于生物学、脑科学、生命科学和心理学等学科的发现,将机理变 为 可 计 算 的 模 型 ( Biologymechanism driven)
1950-1960 二者独立并驾齐驱
1960-1970 符号主义:专家系统和知识工程为主流
1970-1980
1990-2015
2016后
符号主义滞步,日本第五代计算机失败,联接主义蓬勃发展
联接主义占据主导;同时模糊逻辑取得重大进展
生物启发的智能一跨模态的信息处理
同济大学软件学院
A little history about AI
人工智能的研究领域划分人工智能
符号智能
计算智能
机器学习
机器感知
图搜索
自动推理
不确定性推理
符号学习
知识工程
神经计算
进化计算
免疫计算
蚁群计算
归纳学习
模式识别
统计学习
深度学习
计算机视觉
语音识别
自然语言处理
图像识别
同济大学软件学院
A little history about AI
人工智能产业发展加速明显自然语言处理(NLP):微软Skype Translator实现同声传译
计算机视觉(CV):格林深瞳的视频监控可智能识别犯罪
计算机视觉(CV):Face++的人脸识别云服务
感知、规划和决策:Google无人驾驶汽车
同济大学软件学院
A little history about AI
人工智能成为世界焦点
人工智能目前已经成为世界各国关注的焦点。2017年7月,中国政府发布了“新一代人工智能发展规划”
人工智能是开启未来智能世界的秘钥,是未来科技发展的战略制高点;谁掌握人工智能,谁就将成为未来核心技术的掌控者
同济大学软件学院
• Gives "computers the ability to learn without being explicitly programmed” (Arthur Samuel in 1959)
• It explores the study and construction of algorithms that can learn from and make predictions on data
• It is employed in a range of computing tasks where designing and programming explicit algorithms with good performance is difficult or unfeasible
What is machine learning?
[1] Samuel, Arthur L., Some Studies in Machine Learning Using the Game of Checkers, IBM Journal of Research and Development, 1959
Arthur Lee Samuel (December 5, 1901 – July 29, 1990)
同济大学软件学院
Supervised VS Unsupervised
• Supervised learning– It will infer a function from labeled training data – The training data consists of a set of training examples– Each example is a pair consisting of an input object (typically a vector) and a desired output value (also called the supervisory signal)
• Unsupervised learning– Trying to find hidden structure in unlabeled data– Since the examples given to the learner are unlabeled, there is no error or reward signal to evaluate a potential solution
– Such as PCA, K‐means (a clustering algorithm)
同济大学软件学院
About sample
• Attribute (feature), attribute value, label, and example
色泽,根蒂,敲声 {好瓜,坏瓜}features labels
{青绿,蜷缩,浊响:好瓜}feature values label value
one example
同济大学软件学院
Training, testing, and validation
• Training sample and training set
1 1 2 2( , ), ( , ),..., ( , )m mD y y y x x xA training set comprising m training samples,
where is the feature vector of ith sample and is its label
1 2( , ,..., )i i i idx x x x iy
By training, our aim is to find a mapping,
based on D:f
If comprises discrete values, such a prediction task is called “classification”; if it comprises real numbers, such a prediction task is called “regression”
同济大学软件学院
• Training sample and training set• Test set
– A test set is a set of data that is independent of the training data, but that follows the same probability distribution as the training data
– Used only to assess the performance of a fully specified classifier
Training, testing, and validation
同济大学软件学院
• Training sample and training set• Test set• Validation set
– In order to avoid overfitting, when any classification parameter needs to be adjusted, it is necessary to have a validation set; it is used for model selection
– The training set is used to train the candidate algorithms, while the validation set is used to compare their performances and decide which one to take
Training, testing, and validation
同济大学软件学院
• Overfitting– It occurs when a statistical model describes random error or noise instead of the underlying relationship
– It generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations
– A model that has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data
Overfitting, Generalization, and Capacity
同济大学软件学院
• Overfitting• Generalization
– Refers to the performance of the learned model on new, previously unseen examples, such as the test set
Overfitting, Generalization, and Capacity
同济大学软件学院
• Overfitting• Generalization• Capacity
– Measures the complexity, expressive power, richness, or flexibility of a classification algorithm
– Ex, DCNN (deep convolutional neural networks) is powerful since its capacity is very large
* ,y b x *1 1 2 2 ,y b x x
10*
1i i
iy b x
higher capacity
Overfitting, Generalization, and Capacity
同济大学软件学院
higher capacity
Overfitting, Generalization, and Capacity
同济大学软件学院
Performance Evaluation
Given a sample set (training, validation, or test)
1 1 2 2( , ), ( , ),..., ( , )m mD y y y x x x
To assess the performance of the learner f, we need to comparethe prediction and its ground‐truth label y( )f x
For regression task, the most common performance measure is MSE (mean squared error),
2
1
1; ( )m
i ii
E f D f ym
x
同济大学软件学院
• Error rate– The ratio of the number of misclassified samples to the total number of samples
• Accuracy– It is derived from the error rate
Performance Evaluation (for classification)
1
1; ( )1m
i ii
E f D f ym
x
1
1; ( ) 1 ( ; )1m
i ii
acc f D f y E f Dm
x
同济大学软件学院
• Precision and Recall
Performance Evaluation (for classification)
Ground truth Predictionpositive negative
positive True Positive (TP) False Negative (FN)negative False Positive (FP) True Negative (TN)
TPprecisionTP FP
TPrecallTP FN
同济大学软件学院
• Precision and Recall– Often, there is an inverse relationship between precision and recall, where it is possible to increase one at the cost of reducing the other
– Usually, PR‐curve is not monotonic
Performance Evaluation (for classification)
同济大学软件学院
• Precision‐recall should be used together; it is meaningless to use only one of them
• However, in many cases, people want to know explicitly which algorithm is better; we can use F‐measure
Performance Evaluation (for classification)
2
2
(1 )( )
P RFP R
同济大学软件学院
• To derive a single performance measure
Performance Evaluation (for classification)
1 1 2 2( , ), ( , ),..., ( , )n nP R P R P R
Varying threshold, we can have a series of (P, R) pairs,
1
1 n
macro ii
P Pn
1
1 n
macro ii
R Rn
2
1 2macro macro
macromacro macro
P RF
P R
Then,
同济大学软件学院
• Problem definition– It is the problem in machine learning where the total number of a class of data is far less than the total number of another class of data
– This problem is extremely common in practice• Why is it a problem?
– Most machine learning algorithms work best when the number of instances of each classes are roughly equal
– When the number of instances of one class far exceeds the other, problems arise
Class‐imbalance Issue
同济大学软件学院
• How to deal with this issue?– Modify the cost function– Under‐sampling, throwing out samples from majority classes
– Oversampling, creating new virtual samples for minority classes» Just duplicating the minority classes could lead the classifier to overfitting to a few examples
» Instead, use some algorithm for oversampling, such as SMOTE (synthetic minority over‐sampling techniqe)[1]
Class‐imbalance Issue
[1] N.V. Chawla et al., SMOTE: Synthetic Minority Over‐sampling Technique, J. Artificial Intelligence Research 16: 321‐357, 2002
同济大学软件学院
• Minority oversampling by SMOTE[1]
Class‐imbalance Issue
[1] N.V. Chawla et al., SMOTE: Synthetic Minority Over‐sampling Technique, J. Artificial Intelligence Research 16: 321‐357, 2002
(n.feats ‐ c.feats)
同济大学软件学院
Outline
• Basic concepts• Linear model
– Linear regression– Logistic regression– Softmax regression
• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• DCNN for object detection
同济大学软件学院
Linear regression
• Our goal in linear regression is to predict a target continuous value y from a vector of input values ; we use a linear function h as the model
• At the training stage, we aim to find h(x) so that we have for each training sample
• We suppose that h is a linear function, so
dRx
( )i ih yx
1( , ) ( ) ,T d
bh b R x x Rewrite it, ' ',
1b
xx
'' ' '( )T T+b = hx x x
Later, we simply use 1 1 1 1( ) , ,d dTh = R R x x x
( , )i iyx
同济大学软件学院
Linear regression
• Then, our task is to find a choice of so that is as close as possible to
( )ih x
2
1
1( )2
mT
i ii
J y
x
iy
The cost function can be written as,
Then, the task at the training stage is to find
2*
1
1arg min2
mT
i ii
y
x
Here we use a more general method, gradient descentmethod
For this special case, it has a closed‐form optimal solution
同济大学软件学院
Linear regression
• Gradient descent – It is a first‐order optimization algorithm– To find a local minimum of a function, one takes steps proportional to the negative of the gradient of the function at the current point
– One starts with a guess for a local minimum of and considers the sequence such that
0 ( )J
1 |: ( )nn n J
where is called as learning rate
同济大学软件学院
Linear regression
• Gradient descent
同济大学软件学院
Linear regression
• Gradient descent
同济大学软件学院
Linear regression
• Gradient descent
Repeat until convergence ( will not reduce anymore){
}1 |: ( )
nn n J
( )J
GD is a general optimization solution; for a specific problem, the key step is how to compute gradient
同济大学软件学院
Linear regression
• Gradient of the cost function of linear regression
2
1
1( )2
mT
i ii
J y
x
1
2
1
( )
( )( )
( )
d
J
JJ
J
The gradient is,
where, 1
( ) ( )m
i i ijij
J h y x
x
同济大学软件学院
Linear regression
• Some variants of gradient descent– The ordinary gradient descent algorithm looks at every sample in the entire training set on every step; it is also called as batch gradient descent
– Stochastic gradient descent (SGD) repeatedly run through the training set, and each time when we encounter a training sample, we update the parameters according to the gradient of the error w.r.t that single training sample only
Repeat until convergence{
for i = 1 to m (m is the number of training samples){
}}
1 : Tn n n i i iy x x
同济大学软件学院
Linear regression
• Some variants of gradient descent– The ordinary gradient descent algorithm looks at every sample in the entire training set on every step; it is also called as batch gradient descent
– Stochastic gradient descent (SGD) repeatedly run through the training set, and each time when we encounter a training sample, we update the parameters according to the gradient of the error w.r.t that single training sample only
– Minibatch SGD: it works identically to SGD, except that it uses more than one training samples to make each estimate of the gradient
同济大学软件学院
Linear regression
• More concepts– m Training samples can be divided into Nminibatches– When the training sweeps all the batches, we say we complete one epoch of training process; for a typical training process, several epochs are usually required
epochs = 10;numMiniBatches = N; while epochIndex< epochs && not convergent{
for minibatchIndex = 1 to numMiniBatches {update the model parameters based on this minibatch
}}
同济大学软件学院
Outline
• Basic concepts• Linear model
– Linear regression– Logistic regression– Softmax regression
• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• DCNN for object detection
同济大学软件学院
Logistic regression
• Logistic regression is used for binary classification• It squeezes the linear regression into the range (0, 1) ; thus the prediction result can be interpreted as probability
• At the testing stage
T x
1( )1 exp( )Th
xx
Function is called as sigmoid or logisticfunction
1( )1 exp( )
zz
The probability that the testing sample x is positive is represented as
The probability that the testing sample x is negative is represented as 1- ( )h x
同济大学软件学院
Logistic regression
The shape of sigmoid function
One property of the sigmoid function
' ( ) ( )(1 ( ))z z z
Can you verify?
同济大学软件学院
Logistic regression
• The hypothesis model can be written neatly as
• Our goal is to search for a value so that is large when x belongs to “1” class and small when xbelongs to “0” class
1( | ; ) ( ) 1 ( )y yP y h h x x x
Thus, given a training set with binary labels , we want to maximize,
( , ) : 1,...,i iy i mx
11
( ) 1 ( )i im
y yi i
i
h h
x x
( )h x
Equivalent to maximize,
1
log ( ) (1 ) log 1 ( )m
i i i ii
y h y h
x x
同济大学软件学院
Logistic regression
• Thus, the cost function for the logistic regression is (we want to minimize),
To solve it with gradient descent, gradient needs to be computed,
1
( ) log ( ) (1 ) log 1 ( )m
i i i ii
J y h y h
x x
1
( ) ( )m
i i ii
J h y
x x
同济大学软件学院
Outline
• Basic concepts• Linear model
– Linear regression– Logistic regression– Softmax regression
• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• DCNN for object detection
同济大学软件学院
Softmax regression
• Softmax operation– It squashes a K‐dimensional vector z of arbitrary real values to a K‐dimensional vector of real values in the range (0, 1). The function is given by,
– Since the components of the vector sum to one and are all strictly between 0 and 1, they represent a categorical probability distribution
( ) z
1
exp( )( )
exp( )
jj K
kk
zz
z
( ) z
同济大学软件学院
Softmax regression
• For multiclass classification, given a test input x, we want our hypothesis to estimate for each value k=1,2,…,K
( | )p y k x
同济大学软件学院
Softmax regression
• The hypothesis should output a K‐dimensional vector giving us K estimated probabilities. It takes the form,
1
2
1
exp( 1| ; )
exp( 2 | ; ) 1( )exp
( | ; )exp
T
T
K T
jj T
K
p yp y
h
p y K
xx
xxx
xx
x
where ( 1)1 2, ,..., d K
K R
同济大学软件学院
Softmax regression
• In softmax regression, for each training sample we have,
1
exp| ;
exp
Tk i
i i K T
j ij
p y k
xx
x
At the training stage, we want to maximize for each training sample for the correct label k
| ;i ip y k x
同济大学软件学院
Softmax regression
• Cost function for softmax regression
• Gradient of the cost function
1 1
1
exp( ) 1{ }log
exp
Tm K k i
i K Ti kj i
j
J y k
x
x
where 1{.} is an indicator function
1
( ) 1{ } | ;k
m
i i i ii
J y k p y k
x x
同济大学软件学院
Cross entropy
• After the softmax operation, the output vector can be regarded as a discrete probability density function
• For multiclass classification, the ground‐truth label for a training sample is usually represented in one‐hot form, which can also be regarded as a density function
• Thus, at the training stage, we want to minimize
For example, we have 10 classes, and the ith training sample belongs to class 7, then [0 0 0 0 0 010 0 0]iy
( ( ; ), )i ii
dist h y x
How to define dist? Cross entroy is a common choice
同济大学软件学院
Cross entropy
• Information entropy is defined as the average amount of information produced by a probabilistic stochastic source of data
• Cross entropy can measure the difference between two distributions
• For multiclass classification, the last layer usually is a softmax layer and the loss is the ‘cross entropy’
( ) ( ) log ( )i ii
H X p x p x
( , ) ( ) log ( )i ii
H p q p x q x
同济大学软件学院
Outline
• Basic concepts• Linear model• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• DCNN for object detection
同济大学软件学院
Neural networks
• It is one way to solve a supervised learning problem given labeled training examples
• Neural networks give a way of defining a complex, non‐linear form of hypothesis , where W and bare the parameters we need to learn from training samples
{ , }( 1,..., )i iy i mx
, ( )W bh x
同济大学软件学院
Neural networks
• A single neuron– x1, x2, and x3 are the inputs, +1 is the intercept term, is the output of this neuron
, ( )W bh x
3
,1
( ) TW b i i
i
h x f W f W x b
x
where is the activation function( )f
同济大学软件学院
Neural networks
• Commonly used activation functions– Sigmoid function
1( )1 exp( )
f zz
z
( )f z
同济大学软件学院
Neural networks
• Commonly used activation functions– Tanh function
( ) tanh( )z z
z z
e ef z ze e
同济大学软件学院
Neural networks
• Commonly used activation functions– Rectified linear unit (ReLU)
( ) max 0,f z z
同济大学软件学院
Neural networks
• Commonly used activation functions– Leaky Rectified linear unit (ReLU)
, 0( )
0.01 ,z if z
f zz otherwise
=
同济大学软件学院
Neural networks
• Commonly used activation functions– Softplus (can be regarded as a smooth approximation to ReLU)
( ) ln 1 zf z e
同济大学软件学院
Neural networks
• A neural network is composed by hooking together many simple neurons
• The output of a neuron can be the input of another• Example, a three layers neural network,
同济大学软件学院
Neural networks
• Terminologies about the neural network– The circle labeled +1 are called bias units– The leftmost layer is called the input layer– The rightmost layer is the output layer– The middle layer of nodes is called the hidden layer
» In our example, there are 3 input units, 3 hidden units, and 1 output unit
– We denote the activation (output value) of unit i in lay l as ( )lia
同济大学软件学院
Neural networks
(2) (1) (1) (1) (1)1 11 1 12 2 13 3 1
(2) (1) (1) (1) (1)2 21 1 22 2 23 3 2
(2) (1) (1) (1) (1)3 31 1 32 2 33 3 3
(3) (2) (2) (2) (2) 1 (2) (2), 1 11 1 12 2 13 3 1( )W b
a f W x W x W x b
a f W x W x W x b
a f W x W x W x b
h a f W a W a W a b
x
同济大学软件学院
Neural networks
• Neural networks can have multiple outputs• Usually, we can add a softmax layer as the output layer to perform multiclass classification
同济大学软件学院
Neural networks
• At the testing stage, given a test input x, it is straightforward to evaluate its output
• At the training stage, given a set of training samples, we need to train W and b– The key problem is how to compute the gradient– Backpropagation algorithm
同济大学软件学院
Neural networks
• Backpropagation– A common method of training artificial neural networks and used in conjunction with an optimization method such as gradient descent
– Its purpose is to compute the partial derivative of the loss to each parameter (weights)
– neural nets will be very large: impractical to write down gradient formula by hand for all parameters
– recursive application of the chain rule along a computational graph to compute the gradients of all parameters
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Neural networks
• Backpropagation
同济大学软件学院
Outline
• Basic concepts• Linear model• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• CNN for object detection
同济大学软件学院
A little history about deep learning
深度学习
2006年,Hinton和他的学生Salakhutdinov在《科学》上发表了一篇文章,开启了深度学习在学术界和工业界的浪潮。Reducing the Dimensionality of Data with Neural Networks
文章提出了深层网络训练中梯度消失问题的解决方案:无监督预训练对权值进行初始化+有监督训练微调
2012年,Hinton课题组为了证明深度学习的潜力,首次参加ImageNet图像识别比赛,其通过构建的CNN网络AlexNet一举夺得冠军,且碾压第二名(SVM方法)的分类性能。也正是由于该比赛,CNN吸引到了众多研究者的注意
同济大学软件学院
A little history about deep learning
深度学习的发展
提出Inception结构
GoogLeNet
提出残差学习框架网络层次更深
ResNet
网络更深,使用更多的层所有卷积层用同样大小的卷积核
VGG-Net
网络加深解决过拟合多GPU运算
AlexNet
网络结构比较完整,包括卷积层、pooling层、全连接层。被认为是CNN的鼻祖
LeNet
有效解决梯度消失问题大幅度减少参数数量
DenseNet
1998
2012
2014 2016
2014 2017
2017
针对移动端低功耗设备而设计的更为高效的网络
ShuffleNet、MobileNet
针对移动设备,降低深度网络的计算量
PeleeNet
2018
同济大学软件学院
A little history about deep learning
深度学习的应用领域
语音识别人脸认证 图像分类
自动驾驶
生命科学 游戏博弈
DeepLearning
语言翻译
同济大学软件学院
A little history about deep learning
深度学习为什么有效
开源的计算平台Caffe、Tensorflow、Pytorch 等
计算力是引擎GPU服务器、集群
数据就是燃料如ImageNet图像分类数据集等百万张图片
同济大学软件学院
A little history about deep learning
深度学习助力人工智能
深度学习的“层次化”学习思想与人类视觉认知机理高度适应
1958 年 , 约 翰 霍 普 金 斯 大 学David Hubel和Torsten Wiesel发现人的视觉系统的信息处理是分级的,人对物品的识别可能是一个不断迭代不断抽象的过程。
这一发现是神经科学与认知领域的重大突破,促进了人工智能领域以后五十年的发展
同济大学软件学院
A little history about deep learning
深度学习助力人工智能
基础特征提取
中层特征提取
高层特征提取 分类器
像素
边缘
部件
物体
同济大学软件学院
A little history about deep learning
深度学习助力人工智能
人工特征提取
学习特征
简单训练分类器
学习分类器
传统人工智能
深度学习
与传统人工智能相比,深度学习的最大特点就在于
能够自动学习与任务相适应的特征
同济大学软件学院
A little history about deep learning
深度学习与人工智能
人工智能
机器学习
深度学习
人工智能是一个非常广泛的问题,机器学习是解决这类问题的一个重要手段。深度学习则是机器学习的一个分支。在很多人工智能问题上,深度学习的方法突破了传统机器学习方法的瓶颈,推动了人工智能领域的发展
同济大学软件学院
When you know these terminologies, you are an expert of DL
Conv layerPoolingPaddingStrideReluFully connectionBatch normalizationSkip
LeNetAlexNetNINGoogLeNetVGGNetResNetDenseNetMobileNetShuffleNetPeleeNet
RCNNFast-RCNNFaster-RCNNSPPNetMaskNetSSDYoloYoloV2YoloV3Pelee-SSD
CaffeCaffe2TensorFlowMatConvNetTheanoTorchPyTorchKerasMXNetDIGITSTensorRT
同济大学软件学院
When you know these terminologies, you are an expert of DL
GTX980GTX980TiGTX1080GTX1080TiTitanXTitanXPTesla K20Tesla K40Tesla K80Jetson TX1Jetson TX2PX2
ImageNetVOC2007VOC2012MS COCOKittiCityScapes
Gradient vanishingAdversarial samples
Geoffrey HintonAlex KrizhevskyYoshua BengioYann LeCunIan GoodfellowKaiming HeRoss GirshickJoseph Redmon
SYNTHIAVirtual KittiURSAVIPERsynMTGrand Theft Auto V
同济大学软件学院
Convolutional neural network
• Specially designed for data with grid‐like structures (LeCun et al. 98)– 1D grid: sequential data– 2D grid: image– 3D grid: video, 3D image volume
• Beat all the existing computer vision technologies on object recognition on ImageNet challenge with a large margin in 2012
同济大学软件学院
Convolutional neural network
• Something you need to know about DCNN– Traditional model for PR: fixed/engineered features + trainable classifier
– For DCNN: it is usually an end‐to‐end architecture; learning data representation and classifier together
– The learned features from big datasets are transferable– For training a DCNN, usually we use a fine‐tuning scheme– For training a DCNN, to avoid overfitting, data augmentation can be performed
同济大学软件学院
Convolutional neural network
• Problems of fully connected networks– Every output unit interacts with every input unit– The number of weights grows largely with the size of the input image
– Pixels in distance are less correlated
同济大学软件学院
Convolutional neural network
• Problems of fully connected networks
同济大学软件学院
Convolutional neural network
• One simple solution is locally connected neural networks– Sparse connectivity: a hidden unit is only connected to a local patch (weights connected to the patch are called filter or kernel)
– It is inspired by biological systems, where a cell is sensitive to a small sub‐region of the input space, called a receptive field; Many cells are tiled to cover the entire visual field
同济大学软件学院
Convolutional neural network
• One simple solution is locally connected neural networks
同济大学软件学院
Convolutional neural network
• One simple solution is locally connected neural networks– The learned filter is a spatially local pattern– A hidden node at a higher layer has a larger receptive field in the input
– Stacking many such layers leads to “filters” (not anymore linear) which become increasingly “global”
同济大学软件学院
Convolutional neural network
• The first CNN– LeNet[1]
[1] Y. LeCun et al., Gradient‐based Learning Applied to Document Recognition, Proceedings of the IEEE, Vol. 86, pp. 2278‐2324, 1998
同济大学软件学院
Convolutional neural network
Yann LeCun (1960-)New York UniversityFacebook Artificial Intelligence ResearchA founding father of convolutional nets
(His name is a little difficult to pronounce)
同济大学软件学院
Convolutional neural network
He gives himself a Chinese name, 杨立昆 (Mar. 2017)
同济大学软件学院
Convolutional neural network
• Convolution– Computing the responses at hidden nodes is equivalent to convoluting the input image x with a learned filter w
同济大学软件学院
Convolutional neural network
• Downsampled convolution layer (optional)– To reduce computational cost, we may want to skip some positions of the filter and sample only every s pixels in each direction. A downsampled convolution function is defined as
– s is referred as the stride of this downsampled convolution– Also called as strided convolution
( , ) ( * )[ , ]net i j i s j s x w
同济大学软件学院
Convolutional neural network
• Multiple filters– Multiple filters generate multiple feature maps– Detect the spatial distributions of multiple visual patterns
同济大学软件学院
Convolutional neural network
• 3D filtering when input has multiple feature maps
同济大学软件学院
Convolutional neural network
• Convolutional layer
同济大学软件学院
Convolutional neural network
• To the convolution responses, we then perform nonlinear activation– ReLU– Tanh– Sigmoid– Leaky ReLU– Softplus
同济大学软件学院
Convolutional neural network
• Local contrast normalization (optional)– Normalization can be done within a neighborhood along both spatial and feature dimensions
同济大学软件学院
Convolutional neural network
• Then, we perform pooling– Max‐pooling partitions the input image into a set of rectangles, and for each sub‐region, outputs the maximum value
– Non‐linear down‐sampling– The number of output maps is the same as the number of input maps, but the resolution is reduced
– Reduce the computational complexity for upper layers and provide a form of translation invariance
– Average pooling can also be used
同济大学软件学院
Convolutional neural network
• Then, we perform pooling
同济大学软件学院
Convolutional neural network
• Typical architecture of CNN– Convolutional layer increases the number of feature maps– Pooling layer decreases spatial resolution– LCN and pooling are optional at each stage
同济大学软件学院
Convolutional neural network
• Typical architecture of CNN
同济大学软件学院
Convolutional neural network
• Typical architecture of CNN
同济大学软件学院
Convolutional neural network
• Typical architecture of CNN
同济大学软件学院
Convolutional neural network
• Typical architecture of CNN
同济大学软件学院
Convolutional neural network
• Some notes about the CNN layers in most recent net architectures– Spatial pooling (such as max pooling) is not recommended now. It is usually replaced by a strided convolution, allowing the network to learn its own spatial downsampling
– Fully connected layers are not recommended now; instead, the last layer is replaced by global average pooling (for classification problems, the number of feature map channels of the last layer should be the same as the number of classes
同济大学软件学院
Convolutional neural network
• Opensource platforms for CNN– CAFFE official, http://caffe.berkeleyvision.org/– Tensorflow, https://www.tensorflow.org/– Pytorch, www.pytorch.org/– Theano, http://deeplearning.net/software/theano/
同济大学软件学院
Convolutional neural network
• An online tool for network architecture visualization– http://ethereon.github.io/netscope/quickstart.html– Network architecture conforms to the CAFFE prototxt format– The parameter settings and the output dimension of each layer can be conveniently observed
同济大学软件学院
Outline
• Basic concepts• Linear model• Neural network• Convolutional neural network (CNN)• Modern CNN architectures
– AlexNet– NIN– GoogLeNet– ResNet– DenseNet– PeleeNet
• CNN for object detection
同济大学软件学院
AlexNet (NIPS 2012)
• AlexNet: CNN for object recognition on ImageNet challenge– Trained on one million images of 1000 categories collected from the web with two GPU. 2GB RAM on each GPU. 5GB of system memory
– Training lasts for one week– Google and Baidu announced their new visual search engines with the same technology six months after that
– Google observed that the accuracy of their visual search engine was doubled
[1] A. Krizhevsky et al., ImageNet classification with deep convolutional neural networks, in Proc. NIPS, 2012
同济大学软件学院
• ImageNet– http://www.image‐net.org/
AlexNet (NIPS 2012)
同济大学软件学院
• Architecture of AlexNet– 5 convolutional layers and 2 fully connected layers for learning features
– Max‐pooling layers follow first, second, and fifth convolutional layers
AlexNet (NIPS 2012)
同济大学软件学院
• Architecture of AlexNet– The first time deep model is shown to be effective on large scale computer vision task
– The first time a very large scale deep model is adopted– GPU is shown to be very effective on this large deep model
AlexNet (NIPS 2012)
同济大学软件学院
• Main idea of NIN– Conventional convolutional layers uses linear filters followed by a nonlinear activation function to abstract the information within a receptive field
– Instead, NIN uses micro neural networks with more complex structures to abstract the data within the receptive field
– The feature maps are obtained by sliding the micro network over the input in a similar manner as CNN
– Moreover, they use global average pooling over feature maps in the classification layer, which is easier to interpret and less prone to overfitting than traditional fully connected layers
Network In Network (NIN, ICLR 2014)
[1] M. Liu et al., Network in network, in Proc. ICLR, 2014
同济大学软件学院
Network In Network (NIN, ICLR 2014)
• Comparison of linear convolution layer and mlpconv layer – Both the layers map the local receptive field to an output feature vector
– The mlpconv layer maps the input local patch to the output feature vector with a multilayer perceptron (MLP) consisting of multiple fully connected layers with nonlinear activation functions
m n m n
(the last layer of MLP has n nodes)
同济大学软件学院
Network In Network (NIN, ICLR 2014)
The overall structure of NIN. The last layer is the global average pooling
• More about global average pooling – Fully connected layers are prone to overfitting– If there are c classes, the last MLP layer should output c feature maps, one feature map for each corresponding category of the classification task
– Take the average of each feature map to get a c dimensional vector for softmax classification
同济大学软件学院
Network In Network (NIN, ICLR 2014)• NIN can be implemented with conventional convolutional layers
For a mlpconv layer, suppose that the input feature map is of the size , the expected output feature map is of the size 32m m
64m m , the receptive field is ; the mlpconv layer has 2 hidden5 5layers, whose node numbers are 16 and 32, respectively.
How to implement this mlpconv layer with convolutional layers?
同济大学软件学院
GoogLeNet (CVPR 2015)
• Main idea: make the network deeper and wider, while keeping the number of parameters
• Inception module
[1] C. Szegedy et al., Going deeper with convolutions, in Proc. CVPR, 2015
同济大学软件学院
GoogLeNet (CVPR 2015)
• Many Inception modules can stack together to form a very deep network
• GoogLeNet refers to the version the authors submitted for the ILSVRC 2014 competition– This network consists 27 layers (including pooling layers)
同济大学软件学院
ResNet (CVPR 2016 Best Paper)
• What is the problem of stacking more layers using conventional CNNs?– Vanishing gradient, which can hamper the convergence– Accuracy get saturated, and then degraded
[1] K. He et al., Deep residual learning for image recognition, in Proc. CVPR, 2016
同济大学软件学院
ResNet (CVPR 2016 Best Paper)Is there any better way to design deeper networks?Answer: Residual learning
weight layer
weight layer
x
( )xH
Conventional CNN
weight layer
weight layer
x
( )= ( )x x xH F Residual block
xidentity mapping
( )xF
同济大学软件学院
ResNet (CVPR 2016 Best Paper)
• It is easier to optimize the residual mapping (F(x)) than to optimize the original mapping (H(x))
• Identity mapping is implemented by shortcut• A residual learning block is defined as,
= ( ,{ })y x xiF W where x and y are the input and output vectors of the layersF+x is performed by a shortcut connection and element‐wise additionNote: If the dimensions of F and x are not equal (usually caused by changing the numbers of input and output channels), a linear projection Ws (implemented with 1*1 convolution) is performed on x to match the dimensions,
= ( ,{ })y x xi sF W W
同济大学软件学院
ResNet (CVPR 2016 Best Paper)
• It is easier to optimize the residual mapping (F(x)) than to optimize the original mapping (H(x))
• Identity mapping is implemented by shortcut• A residual learning block is defined as,
= ( ,{ })y x xiF W where x and y are the input and output vectors of the layersF+x is performed by a shortcut connection and element‐wise addition
I highly recommend you to take a look the prototxt file of ResNet(https://github.com/KaimingHe/deep‐residual‐networks)
同济大学软件学院
DenseNet (CVPR 2017 Best Paper)
• Highly motivated by ResNet• A DenseNet comprises “dense blocks” and transition layers– Within a dense block, connect all layers with each other in a feed‐forward fashion
– In contrast to ResNet, DenseNet combine features by concatenating them
– The number of output feature maps of each layer is set as a constant within a dense block and is called as “growth rate”
– Between two blocks, there is a transition layer, consisting of batch normalization, convolution, and average pooling
[1] G. Huang et al., Densely connected convolutional networks, in Proc. CVPR, 2017
1 1
同济大学软件学院
DenseNet (CVPR 2017 Best Paper)
• Highly motivated by ResNet• A DenseNet comprises “dense blocks” and transition layers
A sample dense block, whose growth rate is 4
同济大学软件学院
DenseNet (CVPR 2017 Best Paper)
• Highly motivated by ResNet• A DenseNet comprises “dense blocks” and transition layers
A sample DenseNet with three dense blocks
同济大学软件学院
DenseNet (CVPR 2017 Best Paper)
• Highly motivated by ResNet• A DenseNet comprises “dense blocks” and transition layers
• More details about DenseNet design– Bottleneck layers. A convolution layer can be introduced as bottleneck layer before each convolution to reduce the number of input feature maps, and thus to improve computational efficiency
– Compression. If a dense block contains m feature maps, we let the following transition layer generate output feature maps where is referred to as the compression factor
1 13 3
m0 1
同济大学软件学院
PeleeNet (ICLR Workshop 2018)
• MobileNet and ShuffleNet are designed to run on mobiledevices with limited computing power and memory resource
• However, they both depend on depthwise separableconvolution which lacks efficient implementation in most deeplearning frameworks
• PeleeNet– based on conventional convolution instead– follows the innovate connectivity pattern and some of key designprincipals of DenseNet
– achieves a higher accuracy by 0.6% and 11% lower computational costthan MobileNet
– PeleeNet is only 66% of the model size of MobileNet
[1] J. Wang et al., Pelee: A Real‐Time Object Detection System on Mobile Devices,in Proc. ICLR Workshop, 2018
同济大学软件学院
PeleeNet (ICLR Workshop 2018)
• PeleeNet’s design– Two‐Way Dense LayerMotivated by GoogLeNet, use a 2‐way dense layer to get different scalesof receptive fields. One way of the layer uses a small kernel size (3x3),which is good enough to capture small‐size objects. The other way of thelayer uses two stacked 3x3 convolution to learn visual patterns for largeobjects
2‐way dense layer
同济大学软件学院
PeleeNet (ICLR Workshop 2018)
• PeleeNet’s design– Two‐Way Dense Layer– Stem blockThe stem block can effectively improve the feature expression abilitywithout adding computational cost too much ‐ better than other moreexpensive methods, e.g., increasing channels of the first convolutionlayer
stem block
同济大学软件学院
PeleeNet (ICLR Workshop 2018)
• PeleeNet’s design– Two‐Way dense layer– Stem block– Dynamic number of channels in bottleneck layer– Transition layer without compression– Composite FunctionTo improve actual speed, they use the conventional wisdom of postactivation (Convolution ‐ Batch Normalization ‐ Relu) as their compositefunction instead of pre‐activation used in DenseNet
同济大学软件学院
Outline
• Basic concepts• Linear model• Neural network• Convolutional neural network (CNN)• Modern CNN architectures• CNN for object detection
同济大学软件学院
Background
• Detection is different from classification– An image classification problem is predicting the label of an image among the predefined labels; It assumes that there is single object of interest in the image and it covers a significant portion of image
– Detection is about not only finding the class of object but also localizing the extent of an object in the image; the object can be lying anywhere in the image and can be of any size (scale)
同济大学软件学院
Background
Multiple objects detection
同济大学软件学院
Background
• Traditional methods of detection involved using a block‐wise orientation histogram (SIFT or HOG) feature which could not achieve high accuracy in standard datasets such as PASCAL VOC; these methods encode a very low level characteristics of the objects and therefore are not able to distinguish well among the different labels
• Deep learning based methods have become the state‐of‐the‐art in object detection in image; they construct a representation in a hierarchical manner with increasing order of abstraction from lower to higher levels of neural network
同济大学软件学院
Background
• Recent developments of CNN based object detectors– R‐CNN (CVPR 2014)– Fast‐RCNN (ICCV 2015)– Faster‐RCNN (NIPS 2015)– Yolo (CVPR 2016)– SSD (ECCV 1016)– Yolov2 (CVPR 2017)– YoloV3– Pelee‐SSD (ICLR Workshop 2018)
同济大学软件学院
R‐CNN
• Brute‐force idea– One could perform detection by carrying out a classification on different sub‐windows or patches or regions extracted from the image. The patch with high probability will not only the class of that region but also implicitly gives its location too in the image
– One brute force method is to run classification on all the sub‐windows formed by sliding different sized patches (to cover each and every location and scale) all through the image
Quite Slow!!!
同济大学软件学院
R‐CNN
• R‐CNN therefore uses an object proposal algorithm (selective search) in its pipeline which gives out a number (~2000) of TENTATIVE object locations
• These object regions are warped to fixed sized (227X227) regions and are fed to a classification convolutional network which gives the individual probability of the region belonging to background and classes
同济大学软件学院
R‐CNN
同济大学软件学院
Fast‐RCNN
• Compared to RCNN– A major change is a single network with two loss branches pertaining to soft‐max classification and bounding box regression
– This multitask objective is a salient feature of Fast‐RCNN as it no longer requires training of the network independently for classification and localization
同济大学软件学院
Faster‐RCNN
• Compared to Fast‐RCNN– Faster‐RCNN replaces Selective Search with CNN itself for generating the region proposals (called RPN‐region proposal network) which gives out tentative regions at almost negligible amount of time
同济大学软件学院
Yolo (YoloV2)
• Yolo (You Only Look Once)– The major exceptional idea is that it tackles the object detection problem as a regression problem
– A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation
– The whole detection pipeline is a single network– It is extremely fast; With TitanX, it can process ~50 frames/s
同济大学软件学院
Yolo (YoloV2)
同济大学软件学院
Yolo (YoloV2)
• YoloV2– It is a quite recent extension for Yolo– It extends Yolo in the following aspects, batch normalization, high resolution classifier, convolutional with anchor boxes, fine‐grained features, multi‐scale training
– The authors provide both Linux and Windows versions
同济大学软件学院
Thanks for your attention