learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf ·...
TRANSCRIPT
![Page 1: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/1.jpg)
Learning and forgetting in image generation
Luis HerranzComputer Vision Center, UAB
December 2018
BCN.ai
![Page 2: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/2.jpg)
Successful AI in computer visionSiamese
cat
Malignant melanocytic
lesion
Outperform humans!...but• Tons of data• All data in advance• Heavily supervised• Discriminative tasks
![Page 3: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/3.jpg)
Humans learn differently
timeTask 1
ES
Task 3
中文
Task 2
EN
Continual learning
Image generation(“imagination”)
Limited data and supervision: Few-shot learningZero-shot learning
Unsupervised learning
Hey Jan, can you paint a girl with a
pearl earring?
…without forgetting
![Page 4: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/4.jpg)
Continual learning in humans(a.k.a. lifelong/sequential learning)
timeTask 1
ES
Task 3
中文
Task 2
EN
• Reuse of past knowledge (i.e. knowledge transfer, transfer learning)• Learn new skills for new tasks
![Page 5: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/5.jpg)
…and forgetting
time
ENES ENES中文ESEN ENES ENES ENES
HolaHello你好
HolaHello???
Forgetting
![Page 6: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/6.jpg)
Why is continual learning important?Efficient processing of data streams
Learning with privacy (e.g. discard data after processing)
Human-robot interaction Deal with changing environments and problems
time
cat bird dog cat dog
![Page 7: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/7.jpg)
Learning in neural networks
Task 1
Task 2
Task 3
beagle tabby cardinal
![Page 8: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/8.jpg)
Continual learning in neural networks
cardinal
Task 3
?? ??
Catastrophic forgetting
cvTask 1
Task 2
![Page 9: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/9.jpg)
Transfer learning and continual learning
Source task Target taskTask 2
Forgets source task, i.e. catastrophic forgetting (who cares?)
Forgets task 1(big deal!!)
Transfer+adaptation Continual learning
Task 1
Continual learning =transfer learning – (catastrophic) forgetting
![Page 10: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/10.jpg)
Catastrophic interference and forgetting
Kirkpatric et al., Overcoming catastrophic forgetting in neural networks, PNAS, 2017
θ*A
θ*B
Low errorfor task B
Low errorfor task A
θ1
θ2
Training/fine tuning
θ*A
θ*B
Low errorfor task B
Low errorfor task A
θ1
θ2θFA
Elastic weight consolidation (EWC)
forgets task A
θ1 θ2
Input
Output
cvTask ATask B
![Page 11: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/11.jpg)
Catastrophic interference and forgetting
Liu et al., Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting, ICPR 2018http://www.lherranz.org/2018/08/21/rotating-networks-to-prevent-catastrophic-forgetting
Elastic weight consolidation (EWC)
θ*A
θ*B
θ1
θ2FA
θ
Training/fine tuning
θ*A
θ*B
θ1
θ2
θ’2
θ'1
FA
θ'*B
θ'*A θ
Rotated Elastic weight consolidation (R-EWC)
θ
70%
0%
Acc
ura
cy
1-25 1-50 1-75 1-100Classes
Transfer/fine tuning
CIFAR-100
EWCR-EWC
![Page 12: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/12.jpg)
p(x)
Generative models: networks that imagine
12
Training data Sampling
Learning
Different approaches- Density estimation- Variational autoencoders- Autoregressive models
- Generative adversarial networks (GANs)
p(x)ˆ
(e.g. 64x64x3≈12K dims)
![Page 13: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/13.jpg)
p(x)ˆ
Indirect generative sampling
13
How to learn θ ?fθ(z)
Learning and sampling from a complex distribution directly is very difficult
z=-0.5423
Idea: sample from a simple distribution and learn a transformation
z~N(μ=0,σ=1)
Latentrepresentation fθ(z)
![Page 14: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/14.jpg)
Generative Adversarial Networks (GANs)
14
Goodfellow et al., “Generative Adversarial Networks”, NIPS 2014
Classify fake images vs real images
Generate fake samples to fool the discriminator
ZLatent
representation(random) fθ(z) Backpropagation
Real/fake?
Fake image
Training data(real)
Generator
Discriminator
![Page 15: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/15.jpg)
ConditionalGenerative Adversarial Networks
Generator Latent vector z
real/fake?
Training data
Condition c
Classifier ĉ=dog/cat?
Auxiliary Classifier GAN (AC-GAN)
Discriminator
Odena et al., Conditional Image Synthesis With Auxiliary Classifier GANs, ICML 2017
cat
dog
bird
![Page 16: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/16.jpg)
Generative Adversarial Networks
16
Progressive growing of GANs
Wasserstein GAN (WGAN-GP)
BigGAN (conditional GAN)
![Page 17: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/17.jpg)
Generator
Bedroom Kitchen Church
Conditional image generation
c=bedroom
c=kitchen
c=church
![Page 18: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/18.jpg)
c=bedroom
Catastrophic forgetting
Generator
c=kitchen
c=church
cv
Bedroom(task 1)
Aftertask 1
Aftertask 2
Aftertask 3
cv
Kitchen(task 2)
cv
Church(task 3)
Continual learning in image generation
![Page 19: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/19.jpg)
Sequential learning for imagegeneration
c=0 c=1 c=2 c=3 c=4 c=5 c=6 c=7 c=8 c=9MNIST 10 categories (10 tasks)
LSUN 4 categories (4 tasks)c=bedroom c=kitchen c=church c=tower
Go to http://www.lherranz.org/2018/10/29/mergans to play the videos
![Page 20: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/20.jpg)
Memory Replay GANs (MeRGANs)
Generatorz
c
zGenerator
c
Replay generator
Not trained:- Remembers
previous task- Prevents
forgetting
cv
Kitchen(task 2)
cv
Bedroom(task 1)
![Page 21: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/21.jpg)
GAN with EWC
z
c=birdreal/fake?
Training data task 3
Discriminator
church
Seff et al., Continual Learning in Generative Adversarial Nets, arxiv 2017
Current gen. (task 3)
Previous gen. (after task 2)
Initialize
Fisher Informationmatrix (gen)
Estimate F(G)t=2
LEWC
![Page 22: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/22.jpg)
MeRGAN-JTR: joint training w/ replay
Extended Training Data
church
Replay gen. (after task 2)z
c=bed,kitchen
Step 1: replay previous tasks
z
c=bed/kitchen/church
real/fake?Discrim.
Current gen. (task 3)
Classifier ĉ=bed/kitchen/church?
Step 2: joint training
kitchen
bed
Replayed data
Task 3
C. Wu et al., “Memory Replay GANs: learning to generate images from new categories without forgetting”, NeurIPS 2018
![Page 23: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/23.jpg)
MeRGAN-RA: replay alignment
Step 1: replay previous tasks and align
Current gen. (task 3)
c=catc=dog
Previous gen. (after task 2)
LALIGN
zWe can do pixelwise alignment
because for given z and coutput is deterministic
(thanks conditional GAN)
![Page 24: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/24.jpg)
MeRGAN-RA: replay alignment
Current gen. (task 3)
Step 2: learning new task
Training data task 3
bird
real/fake?Discrim.
zc=bird
![Page 25: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/25.jpg)
Digit generation (MNIST)
C. Wu et al., “Memory Replay GANs: learning to generate images from new categories without forgetting”, NeurIPS 2018
c=0 c=1 c=2 c=3 c=4 c=5 c=6 c=7 c=8 c=9MNIST 10 categories (10 tasks)
MeRGANJTR
MeRGANRA
EWC
SFT
Go to http://www.lherranz.org/2018/10/29/mergans to play the video
![Page 26: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/26.jpg)
Scene generation (LSUN)
4 tasks (4 categories) 18-layer ResNet generator
c=bedroom
c=kitchen
c=church
c=tower
Task 1 Task 2 Task 3 Task 4Sequential fine tuning MeRGAN-JTR
Task 1 Task 2 Task 3 Task 4MeRGAN-RA
Task 1 Task 2 Task 3 Task 4
Different bedrooms! Same bedroom!
Remembers the category
Remembers the instance
C. Wu et al., “Memory Replay GANs: learning to generate images from new categories without forgetting”, NeurIPS 2018
![Page 27: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/27.jpg)
Visualizing catastrophic interference
4 tasks (4 categories) 18-layer ResNet generator
Sampling bedrooms
Task 1 Task 2 Task 3 Task 4
Learning church interferes with remembering bedroom
c=bedroom c=kitchen c=church c=tower
![Page 28: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/28.jpg)
Go to http://www.lherranz.org/2018/10/29/mergansto play the video
![Page 29: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/29.jpg)
t-SNE visualizations (MNIST)
Generating digit 0 (i.e. first task) after learning 10 tasks
Samples from MeRGANsoverlap with real data
SFT
RealEWC
MeRGAN-JTR
MeRGAN-RA
C. Wu et al., “Memory Replay GANs: learning to generate images from new categories without forgetting”, NeurIPS 2018
![Page 30: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/30.jpg)
Learning and forgetting in t-SNE (MNIST)
Generating digit 0 (i.e. task 1)
C. Wu et al., “Memory Replay GANs: learning to generate images from new categories without forgetting”, NeurIPS 2018
![Page 31: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/31.jpg)
Collaborators
Chenshen Wu Xialei Liu Yaxing Wang Marc Masana
Joost van de Weijer
Bogdan Raducanu
Antonio López
Andrew Bagdanov
![Page 32: Learning and forgetting in image generationlherranz.org/local/talks/mergans_bcnai_201812.pdf · Task 3 中文 Task 2 EN ... Idea: sample from a simple distribution and learn a transformation](https://reader036.vdocuments.pub/reader036/viewer/2022071213/6037e7f6ee39f333447038a5/html5/thumbnails/32.jpg)
www.cvc.uab.es/[email protected]
More details at http://www.lherranz.org/category/continual-learningMeRGANs, NeurIPS 2018, https://arxiv.org/abs/1809.02058
R-EWC, ICPR 2018 https://arxiv.org/abs/1802.02950
Thank you!