improving variational inference with inverse autoregressive flow
TRANSCRIPT
ImprovingVariational InferencewithInverseAutoregressiveFlow
Jan.19,2017
TatsuyaShirakawa ([email protected])
Diederik P.Kingma (OpenAI)TimSalimans (OpenAI)Rafal Jozefowics (OpenAI)XiChen(OpenAI)IlyaSutskever (OpenAI)MaxWelling(UniversityofAmsterdam)
1
Variational Autoencoder (VAE)
log 𝑝 𝒙
≥
𝔼( 𝒛|𝒙 log 𝑝 𝒙, 𝒛 − log 𝑞(𝒛 |𝒙)∥
log 𝑝 𝒙 − 𝐷23 𝑞 𝒛|𝒙 ∥ 𝑝 𝒛 𝒙∥
𝔼( 𝒛|𝒙 log 𝑝 𝒙 𝒛 − 𝐷23 𝑞 𝒛|𝒙 ∥ 𝑝 𝒛
=: ℒ 𝒙; 𝜽
Modelz ~ p(z;η)x ~ p(x|z;η)
Optimization
maximize𝜼
1𝑁B log 𝑝 𝒙𝒏; 𝜼
D
EFG
Inference Modelz ~ q(z|x;ν)
Optimization
maximize𝜽F(𝜼,𝝂)
1𝑁Bℒ 𝒙𝒏; 𝜽
D
EFG
ELBO
𝒘𝒊𝒕𝒉𝜽 = 𝝁, 𝝂
P(z|x;μ*)
𝑫𝑲𝑳(𝒒 ∥ 𝒑)q(z|x;ν*)
P(z|x;μ)
q(z|x;ν)
2
Requirementsfortheinferencemodelq(z|x)
ComputationalTractability1. Computationallycheaptocomputeanddifferentiate2. Computationallycheaptosamplefrom3. Parallelcomputation
Accuracy4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)
P(z|x;μ*)
𝑫𝑲𝑳(𝒒 ∥ 𝒑)
q(z|x;ν*)
P(z|x;μ)
q(z|x;ν)
3
PreviousDesignsofq(z|x)
BasicDesigns- DiagonalGaussianDistribution- FullCovarianceGaussianDistribution
DesignsbasedonChangeofVariables- Nice
L.Dinh etal.,“Nice:non-linearindependentcomponentsestimation”,2014
- NormalizingFlowD.J.Rezende etal.,“Variational inferencewithnormalizingflows”,ICML2015
DesignsbasedonAddingAuxiliaryVariables- HamiltonianFlow/HamiltonianVariational Inference
T.Salimans etal.,”MarkovchainMonteCarloandvariational inference:Bridgingthegap”,2014
4
Diagonal/FullCovarianceGaussianDistribution
Diagonal:Efficientbutnotflexible𝑞 𝒛 𝒙 = ΠU𝑁 𝒛𝒊|𝜇U 𝒙 , 𝜎U 𝒙
FullCovariance:NotEfficientandnotflexible(unimodal)𝑞 𝒛 𝒙 = 𝑁 𝒛|𝝁 𝒙 , 𝚺 𝒙
1. Computationallycheaptocomputeanddifferentiate ✓ / ✗2. Computationallycheaptosamplefrom ✓ / ✗3. Parallelcomputation ✓ / ✗4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✗
5
ChangeofVariablesbasedmethods
Transoform𝑞 𝑧Z 𝑥 tomakemorepowerful distribution𝑞 𝑧\ 𝑥 viasequentialapplicationofchangeofvariables
𝒛𝒕 = 𝑓 𝒛𝒕_𝟏
𝑞 𝒛𝒕 𝒙 = 𝑞 𝒛𝒕_𝟏 𝒙 det𝑑𝑓 𝒛𝒕_𝟏𝑑𝒛𝒕_𝟏
_G
⇒ log 𝑞 𝒛𝑻 𝒙 = log 𝑞 𝒛𝟎 𝒙 −Blog det𝑑𝑓 𝒛𝒕_𝟏𝑑𝒛𝒕_𝟏
�
^
• NiceL.Dinh etal.,“Nice:non-linearindependentcomponentsestimation”,2014
• NormalizingFlowD.J.Rezende etal.,“Variational inferencewithnormalizingflows”,ICML2015
6
NormalizingFlow
Transformationvia𝒛𝒕 = 𝒛𝒕_𝟏 + 𝒖𝒕𝑓 𝒘𝒕
\𝒛𝒕_𝟏 + 𝑏^KeyFeatures- Determinantsarecomputable
Drawbacks- Informationgoesthroughsinglebottleneck
1. Computationallycheaptocomputeanddifferentiate ✓2. Computationallycheaptosamplefrom ✓3. Parallelcomputation ✗4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✗
singlebottleneck
⊕
𝒛𝒕_𝟏
𝒛𝒕
𝒘𝒕𝑻𝒛𝒕 + 𝑏^
𝒖𝒕𝑓 𝒘𝒕𝑻𝒛𝒕 + 𝑏^
7
HamiltonianFlow/HamiltonianVariational Inference
ELBOwithauxiliaryvariablesylog 𝑝 𝒙 ≥ log 𝑝 𝒙 − 𝐷23 𝑞 𝒛|𝒙 ∥ 𝑝 𝒛 𝒙 − 𝐷23 𝑞 𝒚 𝒙, 𝒛 ∥ 𝑟 𝒚 𝒙, 𝒛 =: ℒ 𝒙
Drawing(y,z)viaHMC𝑦^, 𝑧^ ~𝐻𝑀𝐶 𝑦^, 𝑧^|𝑦^_G, 𝑧^_G
KeyFeatures- Capabilitytosamplefromexactposterior
Drawbacks- LongmixingtimeandlowerELBO
1. Computationallycheaptocomputeanddifferentiate ✗2. Computationallycheaptosamplefrom ✗3. Parallelcomputation ✗4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✓
8
Nice
Transformonlyhalfofzateachsteps
𝒛𝒕 = 𝒛𝒕𝜶, 𝒛𝒕𝜷 = 𝒛𝒕_𝟏𝜶 , 𝒛𝒕_𝟏
𝜷 + 𝑓 𝒙, 𝒛𝒕_𝟏𝜶 ,KeyFeatures- DeterminantoftheJacobiandet uvw 𝒛𝒕x𝟏
u𝒛𝒕x𝟏isalways1
Drawbacks- Limitedformoftransformation- lessaccuratepowerfulthanNormalizingFlow(Next)
1. Computationallycheaptocomputeanddifferentiate ✓2. Computationallycheaptosamplefrom ✓3. Parallel computation ✗4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✗
9
Autoregressive Flow(proposed)
AutoregressiveFlow(𝑑𝜇^,U/𝑑𝑧^,z =𝑑𝜎^,U/𝑑𝑧^,z =0if𝑖 ≤ 𝑗)𝑧^,U = 𝜇^,U 𝒛𝒕,𝟎:𝒊_𝟏 + 𝜎^,U 𝒛𝒕,𝟎:𝒊_𝟏 ⊙ 𝑧^_G,U
Keyfeatures- Powerful- Easytocomputedet 𝜕𝒛𝒕/𝜕𝒛𝒕_𝟏 = ΠU𝜎^,U 𝐳𝐭_𝟏
Drawbacks- Difficulttoparallelize
1. Computationallycheaptocomputeanddifferentiate ✓2. Computationallycheaptosamplefrom ✓3. Parallel computation ✗4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✓
10
InverseAutoregressive Flow(proposed)
InvertingAF(𝝁𝒕, 𝝈𝒕isalsoautoregressive)
𝒛𝒕 =𝒛𝒕_𝟏 − 𝝁𝒕 𝒛𝒕_𝟏
𝝈𝒕 𝒛𝒕_𝟏KeyFeatures- EquallypowerfulasAF- Easytocomputedet 𝜕𝒛𝒕/𝜕𝒛𝒕_𝟏 = 1/ΠU𝜎^,U 𝐳𝐭_𝟏- Parallelizable
1. Computationallycheaptocomputeanddifferentiate ✓2. Computationallycheaptosamplefrom ✓3. Parallelcomputation ✓4. Sufficientlyflexibletomatch
thetrueposteriorp(z|x)✓
11
IAFthroughMaskedAutoencoder (MADE)
Modelingautoregressive𝝁𝒕 and𝝈𝒕 withMADE
• RemovingpathsfromfuturesfromAutoencodersbyintroducingmasks•MADEisaprobabilisticmodel𝑝 𝑥 = ΠU𝑝 𝑥U 𝑥Z:U_G
12
Experiments
IAFisevaluatedonimagegeneratingmodels
ModelsforMNIST- ConvolutionalVAEwithResNet blocks- IAF=2-layerMADE- IAFtransformationsarestackedwithorderingreversedalternately
ModelsforCIFAR-10(verycomplicated)
13
MNIST
14
CIFAR-10
15
IAFin1slide
𝑫𝑲𝑳(𝒒 ∥ 𝒑)
𝒒 𝒛𝑻 𝒙; 𝝂𝑻 𝝂𝑻
𝒑 𝒛 𝒙; 𝝁∗𝒑 𝒛 𝒙; 𝝁
𝒒 𝒛 𝒙; 𝝂𝑻∗
𝒒 𝒛𝒕 𝒙; 𝝂𝒕 𝝂𝒕
𝒒 𝒛𝟎 𝒙; 𝝂𝟎 𝝂𝟎
Autoregressive Flow
Inverse Autoregressive Flow
IAF isü Easy to compute and differentiateü Easy to sample fromü Parallelizableü Flexible
𝒒 𝒛 𝒙; 𝝂𝑻
Wearehiring!http://www.abeja.asia/
https://www.wantedly.com/companies/abeja