Everyone can be van GoghIntroduction to Neural Art

物理三 王昱翔

2016. 11. 08

Reference

• Leon A. Gatys, Alexander S. Ecker, Matthias Bethge, A Neural Algorithm of Artistic Style

• Mark Chang’s slide, http://www.slideshare.net/ckmarkohchang/a-neural-algorithm-of-artistic-style?next_slideshow=1

Outline

• Few examples of Neural Art

• Introduction to CNN

• Implement Neural Art

• Application

• Conclusion

Paintings of van Gogh

Can we paint like van Gogh? (I’m not 9.2%)

Yes! (Use Convolutional Neural Network)

Ref: Mark Chang’s slide

Neural Network

Partial CNN Process

Image(Input)

Convolution Max Pooling

CNN– Convolution

0 0 1 1 0 0

0 1 1 1 1 0

1 1 1 1 1 1

1 1 1 1 1 1

0 0 0.5 0.5 0 0

0 0 0.5 0.5 0 0

0 0 1

0 0 1

0 0 1

Filter_2

Number of filters depends on Users !

0 0 1

0 0.5 0

1 0 0

Filter_1

Find pattern

Input image

Filter -- Element wise matrix multiplication

0 0 1 1 0 0

0 1 1 1 1 0

1 1 1 1 1 1

1 1 1 1 1 1

0 0 0.5 0.5 0 0

0 0 0.5 0.5 0 0

0 0 1

0 0.5 0

1 0 0

0 0 1

0 0 1

0 0 1

2.5

CNN -- convolution 0

1

2

3

4

5

6

w0 = 𝐹11

0 1 2 3 4 56 7 8 9 10 11

0 0 1 1 0 0

0 1 1 1 1 0

Corresponding Index

2.5

w1 = 𝐹12

w2 = 𝐹13

w6 = 𝐹21

0 0 1 1 0 0

0 1 1 1 1 0

1 1 1 1 1 1

1 1 1 1 1 1

0 0 0.5 0.5 0 0

0 0 0.5 0.5 0 0

0 0 1

0 0.5 0

1 0 0

0 0 1

0 0 1

0 0 1

2.5

CNN – Max Pooling

2.5 2.5 1.5 1.5

2.5 2.5 2.5 1.5

1.5 1.5 2 2

1 1.5 1.75 1.5

2.5 2.5

1.5 2

0 0 1 1 0 0

0 1 1 1 1 0

1 1 1 1 1 1

1 1 1 1 1 1

0 0 0.5 0.5 0 0

0 0 0.5 0.5 0 0

0 0 1

0 0.5 0

1 0 0

Max pooling

2.5

Repeat Convolution + Max pooling

Input Image

ConvolutionMax pooling

ConvolutionMax pooling

Whole CNN Process

Image ConvolutionMax

Pooling

N times

FlattenFully

ConnectedNeural

Network

Neural Art

Content

Style

Neural Art- Create Content

0.3 0.7 1 0 0.6

0.2 0.8 0.7 0.4 0.4

0.7 0.1 0.2 0.3 0.5

0.8 0 0.3 0.9 0.1

0.5 0.9 0.6 1 0.8

CNN

CNN

Canvas x

Output X

Output P

𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡 =1

2 𝑖,𝑗(𝑋𝑖,𝑗 − 𝑃𝑖,𝑗)

2

Gradient Descent (Back Propagation)

Photo p

Canvas x: Our paintings

CNN: Already trained byother people

CNN

Neural Art – Create Style

0.3 0.7 1 0 0.6

0.2 0.8 0.7 0.4 0.4

0.7 0.1 0.2 0.3 0.5

0.8 0 0.3 0.9 0.1

0.5 0.9 0.6 1 0.8

CNN

Canvas x

Output

Output 𝐿𝑠𝑡𝑦𝑙𝑒 =1

2 𝑖,𝑗(𝑋𝑖,𝑗 − 𝑆𝑖,𝑗)

2

Gradient Descent (Back Propagation)

Style sGram S

Gram X

G

G

∗ 𝐺𝑟𝑎𝑚𝑖,𝑗 = <𝑣𝑖 , 𝑣𝑗>

Gram S_00= <O_s row[0], O_s col[0]>

Update Canvas

• 𝐿𝑡𝑜𝑡𝑎𝑙 = 𝛼𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡 + 𝛽𝐿𝑠𝑡𝑦𝑙𝑒 x = x - 𝜂𝜕𝐿𝑡𝑜𝑡𝑎𝑙

𝜕𝑥

Application -- DeepDream

Ref: Deep Dream

Application -- Prisma

Ref: Prisma

Real time Neural Art

Ref: Fast Style transfer in Tensorflow

Conclusion

• Yeah! Computer can paint by themselves!

• Really? They still need a photo for painting.

Thank You For Your Listening !