최신인공지능기술연구동향과미래전망, 그리고미래를대비한네이버의 AI

Jung-Woo Ha, PhD

Research Head, Clova AI, NAVER & LINE

5 November 2019

Contents

• 최근 AI 연구 및 기술 동향

• AI 기술/서비스의 미래 키워드

• NAVER의 AI – Clova

2

NAVER as a Global Player

3

최근 AI연구및기술동향

Powerful Pretrained Models

5

Backbone Heads

Object detectorInput Output

Classification, Segmentation, Pose, Face, GM, …

Powerful Pretrained Models (Computer Vision)

• ResNet, ResNeXt, WideResNet

• DenseNet, AmoebaNet with Gpipe, and EfficientNet

• InstagramNet

• Squeeze-Excitation

• Optimization

• Regularization

• Data augmentation

• Neural architecture search

6

[Tan & Le. ICML 2019]

Powerful Pretrained Models (NLP)

• W2V [Mikolov et al. 2013]

• GloVe [Pennington et al. 2014]

• ELMo [Peters et al. 2018]

• BERT [Devlin et al. 2018]• RoBERTa [Liu et al. 2019]

• ELECTRA [Anonymous 2019]

• GPTv2 [Radford et al. 2019]

• XLNet [Yang et al. 2019]

• ERNIE 2.0 [Sun et al. 2019]

• LaRva [Lee et al. 2019]

7

[Google AI Blog 2018]

Written by AI (OpenAI GPTv2)

8

How it can?

• More Parameters (Neural Network)

• More Data

• More GPUs

9

AutoML

• AutoML?• Automatic machine learning• Too large search space for finding good models• AutoML Workshop @ ICML

• Hyperparameter Optimization• Hyperband [Li et al. 2016]• BOHB [Falkner et al. 2018]

• Neural Architecture Search (NAS)• NAS with RL [Zoph & Le 2016]• PathNet [Fernando et al. 2017]• DARTS [Liu et al. 2019]• ProxylessNAS [Cai et al. 2019]

• ETC• AutoAugment [Cubuk et al. 2018]• Fast AutoAugment [Lim et al. 2019]

10

Lightweight

• Powerful? Excellent but…..

• Practical?• Many GPU

• Inference

• ROI

• On-device

• Lightweight !!!

11

Lightweight

• Small models for CNN backbones• MobileNet V1 / V2 / V3 [Howard et al. 2017 / Sandler et al. 2018 / Howard et al. 2019]• ShuffleNet V1 / V2 [Zhang et al. 2017 / Ma et al. 2018]• Han et al. 2019

• Distillation• Efficient knowledge transfer-based training for small models• ResNet-152 ResNet-50 // ResNet-50 MobileNet• Hinton 2015• Margin ReLU [Heo et al. 2019]

• Lightweight BERT• DistilBERT [Sanh et al. 2019]• PKD [Sun et al. 2019]• MobileBERT [Anonymous 2019]

• Face: EXTD [Yoo et al. 2019]

• NAS: model size & latency

12

AI 기술/서비스의미래키워드

Current AI

• AlphaGo (2016) AlphaFold (2018) AlphaStar* (2019)

14

AI: Beyond Technology

• Innovation itself

• New services

• New products

• New values: Enhance conventional things• Manufacturing

• Process

• Cost reduction

• Efficiency

• Embedded to enhance

15

Personalized

• AI assistant

• Each user-customized service

• Content

• Advertisement

• Interface: voice, gesture, and thinking

16

Small Data

• Many data but small labeled data

• Few shot learning

• Golden samples

• Active learning

• Self-supervised or semi-supervised

• Unsupervised

• Augmentation

• Generative Models

17

On-Device

• Edge computing – 5G

• On-device inference

• On-device training

• Low power

• Lightweight

• AI Chip

18

https://www.youtube.com/watch?v=RjMS15V_7nQ

Data Privacy

• On-device

• Personal data Sensitive data, Privacy

• Federated learning• https://venturebeat.com/2019/06/03/how-federated-learning-could-shape-the-

future-of-ai-in-a-privacy-obsessed-world/

• Google FL WS: https://sites.google.com/view/federated-learning-2019/presentations

• Nvidia: https://venturebeat.com/2019/10/13/nvidia-uses-federated-learning-to-create-medical-imaging-ai/

• Secure knowledge transfer

19[https://ai.googleblog.com/2017/04/federated-learning-collaborative.html]

Human vs. AI? Human with AI!!!

• Labeling with active learning• Easy data Auto labeling by AI

• Difficult data by Human

• Contact center AI• 단순한 질의는 AI로

• 정신노동에 가까운 응대도 AI로

• 구체적 설명이 필요한 것은 사람담당자가

20

네이버의 AI – Clova 중심으로

NAVER & LINE

22

AI

AI of NAVER & LINE

23

NAVER

Korea No.1 Internet Platform Company

LINE

Global Messenger Platform Company

Based on Japan

Clova

Vision of Clova: AI for Everyone

Innovative AI

Technologies

Practical &

Valuable AI

Services

B2B

B2CBetter World

24

Clova AI Core Technologies to Services

25

Deep

Learning

Speech

NLP

Recomme-

ndation

ML

Platform

Computer

Vision

OCR

SpeechRecognition

Chatbot

FaceSDK

Voice Synthesis

AI ARS

NSML

CutMix: New Robust Data Augmentation

• Data augmentation: Core techs for enhancing backbone performances

• Improvement, robustness, transferability, and computational efficiency

26

[Yun et al. ICCV 2019 (Oral)]

CutMix: New Robust Data Augmentation

27

[Yun et al. ICCV 2019 (Oral)]

Distillation: Student Better Than A Teacher

Our approach: Margin-ReLU and located at pre-ReLU

28

[Heo et al. ICCV 2019]

Distillation: Student Better Than A Teacher

29

[ImageNet-1k] [Other Tasks: Object Detection & Segmentation]

[Heo et al. ICCV 2019]

Lightweight: Change Your Backbone

30

[Classification (ImageNet-1k)]

Models Top-1 err (%)

Params(M)

Flops(G)

MobileNet_v2 28.0 3.51 0.33

ShuffleNet_v2 27.4 3.50 0.30

MNasNet 26.0 4.2 0.32

ClovaAI-A 22.8 3.99 0.75

ClovaAI-B 23.6 3.76 0.42

ResNet-50 24.7 23.5 4.12

ResNet-101 23.6 42.5 7.85

[Detection (MSCOCO)]

Models mAP(%) Params(M)

MobileN_v1 + SSDLite

22.2 5.1

MobileN_v2 + SSDLite

22.1 4.3

MobileN_v3 + SSDLite

22.0 4.97

ClovaAI 23.5 4.8

SSD300 23.2 36.1

YOLOv2 21.6 50.7

Models mAP(%) Params(M)

S3FD+VGG 85.9 22

MobileFaceNet 74.1 1.2

PyramidBox 88.7 22

EXTD(ClovaAI)

86.6 0.16

[Face (MSCOCO)]

• Lightweight backbone as a pretrained model

• Object detector, OCR, FaceSDK, pose estimator, video recognition, etc.

• See the details at DEVIEW 2019

EXTD: EXtremely Tiny face Detector

Recurrent usages of convolution blocks + FPN (os SSD)

31

[Yoo et al. Arxiv 2019]

EXID via EXTD

32

[EXTD: 0.16M] [MobileFaceNet: 1.5M]

0.1초 얼굴 인식

33

Detection in OCR

CRAFT: Character region awareness for text detection

34

[Baek et al. CVPR 2019]

OCR Challenges

35

OCR Services

36

[디지털타임즈, 2019.7.16]

손글씨 폰트 생성

37

SOTA Image Retrieval in Smart Lens

38

[Jeon et al. Arxiv 2019]

Merchandise

(recall@1 vs Alibaba 79%)86%

Video Intelligence

39

Clique – Interactive 3D Avatar

40

Clique – Interactive 3D Avatar

41

HDTS: SOTA Speech Synthesis

42

Required recorded voice data: 10hrs 4hrs 40mins[E. Song et al. ArXiv 2018, Yamamoto et al. Interspeech 2019]

화상 회의록 요약

43

Audio-Visual Speech Enhancement

44

• Speech separation given lip regions in the video[Afouras et al. Interspeech 2018, Chung et al. ICASSP 2019 & Interspeech 2019]

LaRva: Language Representation by Clova

• BigLM based on BERT

• New task definition• GLUE vs. KLUE and JLUE

• New encoding, corpus-level curriculum learning, n-gram masking

• Distributed learning for LaRva

• Fast LaRva

45

LaRva: Language Representation by Clova

• Giant-scale general-purpose language model by improving BERT

• Not Multi-lingual BERT but LaRva

46

[WikiSQL] [KorQuAD]

DUET to AI Call of NAVER Glace

47

NAVER AI @ DEVIEW 2019 with Mr. President

48

Project 1784

49

Global AI R&D Belt

50

NAVER Smart Machine Learning (NSML)

NSML: ML research platform that enables you to focus on your model!!

51

[Sung et al. MLSYS 2017@NIPS 2017]

NAVER Smart Machine Learning (NSML)

52

쉬운 설치와 실행

53

기술보다 사용자 경험 중심의 설계

• AI Researcher, Engineer의 삶에 도움이 되는 설계• AI Researcher, AI Engineer가 더 나은 모델을 만들어 내도록 설계

• 시스템 사용 경험을 통해 Insight를 얻을 수 있도록

• AI Team에 도움이 되는 설계• 문제와 연구자를 Decouple하여 업무 환경 유동성 확보

54

Meeting with NSML

55

효율적/효과적 자원배분

56

리더보드: 데이터/문제 중심 협업의 핵심

57

AutoML

58

AutoML: 연구+프러덕트화 효율성 극대화

실제 OCR 적용 케이스

59

Active Learning 중심의 데이터 레이블링

60

Achievements of AI Research

AI 연구실적 발표: NeurIPS, ICML, ICLR, CVPR, I(E)CCV, ACL, EMNLP, AAAI, IJCAI, ICASSP, Interspeech, KDD, …

61

1 3 5

15

26

2015 2016 2017 2018 2019

Top AI 학회 논문 발표

NSML

7. 풍부한 해커톤 경험

62

Selected Publications of Clova AI in 2019

63

1. Song et al. Hierarchical Context enabled Recurrent Neural Network for Recommendation, AAAI 2019.

2. Park et al. Adversarial Dropout for Recurrent Neural Networks, AAAI 2019.

3. Park et al. Paraphrase Diversification using Counterfactual Debiasing, AAAI 2019.

4. Heo et al. Knowledge Distillation with Adversarial Samples Supporting Decision Boundary, AAAI 2019.

5. Heo et al. Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons, AAAI 2019.

6. Oh et al. Modeling Uncertainty with Hedged Instance Embeddings, ICLR 2019.

7. Gu et al. DialogWAE: Multimodal Response Generation with Conditional Wasserstein Auto-Encoder, ICLR 2019.

8. Lee et al. Large-Scale Answerer in Questioner's Mind for Visual Dialog Question Generation, ICLR 2019.

9. Kim et al. Curiosity-Bottleneck: Exploration By Distilling Task-Specific Novelty, ICML 2019.

10. Chung et al. Perfect match: Improved cross-modal Embeddings for audio-visual synchronisation, ICASSP 2019

11. Moon et al. Domain Mismatch Robust Acoustic Scene Classification Using Channel Information Conversion, ICASSP 2019

12. Baek et al. Character Region Awareness for Text Detection, CVPR 2019.

13. Seo et al. Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index, ACL 2019.

14. Chung et al. Who said that?: Audio-visual speaker diarisation of real-world meeting, Interspeech 2019.

15. Afouras et al. My lips are concealed: Audio-visual speech enhancement through obstructions, Interspeech 2019.

16. Yamamoto et al. Probability Density Distillation with Generative Adversarial Networks for High-quality Parallel Waveform Generation, Interspeech 2019.

17. Hwang et al. Parameter enhancement for MELP speech codec in noisy communication environment, Interspeech 2019.

18. Kim et al. Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation, RECSYS 2019.

19. Yun et al. CutMix: CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features, ICCV 2019.

20. Baek et al. What is wrong with scene text recognition model comparisons? dataset and model analysis, ICCV 2019.

21. Yoo et al. Photorealistic Style Transfer via Wavelet Transforms, ICCV 2019.

22. Heo et al. A Comprehensive Overhaul of Feature Distillation. ICCV 2019.

23. Cho et al. Mixture Content Selection for Diverse Sequence Generation, EMNLP 2019.

24. Chen et al. NL2pSQL: Generating Pseudo-SQL Queries from Under-Specified Natural Language Questions, EMNLP 2019.

25. Kim. Subword Language Model for Query Auto-Completion. EMNLP 2019.

26. Lee et al. Unpaired Sketch-to-Line Translation via Synthesis of Sketches. SIGGRAPH Asia 2019.

Send Your CV Today!

64

clova-jobs@navercorp.com

[Clova AI Github] [Facebook Clova AI Research] [Clova AI Tech Blog]