Chul−Kee Park1, Sungroh Yoon2

1서울대학교의과대학신경외과

2서울대학교공과대학전기정보공학부

자동화부피측정알고리즘과기계학습기법을이용한두개강내뇌수막종의자연사연구

2017 의대-공대학제간융합연구결과발표회

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Contents

2/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Introduction

3/27

…

• Objective: 뇌수막종자연사규명및종양성장모델생성

• 가설: 뇌수막종은 3단계의자연사경과를거침

• Method: 뇌수막종의자동화부피측정알고리즘개발(phase 1까지개발됨)

• Phase 1: Fully automated meningioma segmentation

• Phase 2: volume 측정, Phase 3: 종양성장모델생성

Introduction

Volumetry of meningiomaSegmentation

Integrateresults

종양성장 모델 생성4/27

• 최근 computer vision 분야의 segmentation task는전통적인방법과대비해두

드러지는발전을이룸

• VOC 2012, MS COCO와같은 benchmark dataset이공개되어모델의학습

이용이 (original image와 pixel-wise label이 pair로존재)

• Fully Convolutional Network (FCN)과같은 segmentation에최적화된모델

의제안

• 의료영상 segmentation task의경우일반적인방법론의적용이어려운상황

• 의료영상에적합한 segmentation task를수행하기위해서는그것을위한

deep learning 모델을학습시켜야함

• 의료영상은일반영상과특성이달라단순히 transfer learning을적용하기에

한계가있음

• 의료진이직접만든대량의 pixel-wise label이필요 (heavy annotation effort)

• 이를극복하기위해본연구에서는weakly supvervised approach를적용

Introduction: Meningioma MR data segmentation

5/27

• Object Localization (object detection)

• Fully Supervised Object Localization

• R-CNN, Faster R-CNN and MultipathNet, etc.

• Using location information of region-of-interests (ROIs)

• It requires heavy annotation efforts from human resources

• Weakly Supervised Object Localization

• CAM model [1]

• Only uses image-level labeled datasets to train a network

Introduction: Weakly supervised approach

[1] B. Zhou, et at. “Learning Deep Features for Discriminative Localization.” Computer Vision and Pattern Recognition (CVPR), 2016.

6/27

• Meningioma segmentation pipeline

• ResCAM (Object localization model)

• Image-level labeled data를학습하여영상속종양의대략적인위치를찾아주

는 attention (class activation map)추출

• FCN (Segmentation model)

• ResCAM에서생성된 attention과소량의 pixel-wise labeled data를학습하여

영상속종양을추출

Introduction: Our approach

Meningioma 영상

FCN OutputResCAM

7/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Related work

8/27

• M. Lin, Q. Chen, and S. Yan, “Network In Network,” ICLR 2014.

• In the last convolution layer of CNNs

• Replace the traditional fully connected layers with global average pooling

• Generate one feature map for each corresponding category of the classification

task in the last mlpconv layer

• Take the average of each feature map, and the resulting vector is fed directly in

to the softmax layer

Global Average Pooling

Global average pooling

It is more native to the convolution structure by enforcing correspondences between feature maps and categories.

9/27

• B. Zhou, A. Khosla et al., “Learning Deep Features for Discriminative Localization,”

CVPR 2016.

• CAM represent discriminative image regions used by the CNN to identify that category

• Model architecture

• Global Average Pooling (GAP)

• Fully-connected layer + softmax layer

Class Activation Map (CAM)

10/27

• Activation of unit k in the last convolutional layer at spatial location (𝑥, 𝑦) : 𝑓𝑘(𝑥, 𝑦)

• Global average pooling output 𝐹𝑘: 𝐹𝑘 = 𝑥,𝑦 𝑓𝑘(𝑥, 𝑦)

• Input to the softmax: 𝑆𝑐 = 𝑘𝑤𝑘𝑐𝐹𝑘

• 𝑐: class, 𝑤𝑘𝑐: weight corresponding to class c for unit k

• 𝑤𝑘𝑐 indicates the importance of 𝐹𝑘 for class c

• Output of the softmax for class c, 𝑃𝑐 is given byexp(𝑆𝑐)

𝑐 exp(𝑆𝑐)

• By plugging 𝐹𝑘 = 𝑥,𝑦 𝑓𝑘(𝑥, 𝑦) in to the class score 𝑆𝑐𝑆𝑐 = 𝑘𝑤𝑘

𝑐 𝑥,𝑦 𝑓𝑘(𝑥, 𝑦)

= 𝑥,𝑦 𝑘𝑤𝑘𝑐𝑓𝑘(𝑥, 𝑦)

• Define 𝑀𝑐 as the class activation map for class 𝑐

𝑀𝑐(𝑥, 𝑦) = 𝑘𝑤𝑘𝑐𝑓𝑘(𝑥, 𝑦)

Class Activation Map (CAM)

11/27

• By simply upsampling the class activation map to the size of the input

image, we can identify the image regions most relevant to the particular

category

Class Activation Map (CAM)

Global average

pooling …

Class Activation Map

𝑀𝑐(𝑥, 𝑦) indicates the importance of the activation at spatial grid (𝑥, 𝑦)

12/27

• Long, Jonathan, Evan Shelhamer et al. "Fully convolutional networks for semantic

segmentation," CVPR 2015.

• Reinterpret standard classification convnets as “fully convolutional” networks (FCN)

for semantic segmentation

• Novel architecture: combine information from different layers for segmentation

• Trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic

segmentation

• Inference less than one fifth of a second for a typical image

Fully Convolutional Network (FCN)

13/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Proposed method

14/27

• Attention map + FCN based segmentation

Our approach

(좌): input image, (우): saliency map

Fully Convolutional Network

(좌): input image, (우): Result

CAM extraction

• Extract attention map• ResNet based model

1.

FCN segmentation

• Using original image and attention map for input

2.

Meningioma MR

• Segmentation

3.

Meningioma volumetry

• Integrate segmentation results

4.

15/27

• To produce image-level labeled data, we create annotation tool

• Press an appropriate value (0: normal, 1: normal, 2: too noisy, 3: pass)

• Display three slices (former, current and following image)

Labeling System

16/27

• Resnet기반의CAM model 생성

• Binary classification (영상에종양유무판별)을수행하면서, discriminative

region을추출

• 각환자 sequence 전체에대하여평균을구한뒤,하나의CAM 생성

(false positive를줄이기위해)

ResCAM

17/27

• FCN 기반의 segmentation 사용 (VGG16 base FCN8s model)

• 원본영상과CAM 영상을합쳐 FCN의입력으로사용

• Ch1,2: 원본영상

• Ch3: CAM 영상

FCN-meningioma segmentation

+

18/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Experiment setup

19/27

• Data: T1 Enhanced MR brain data

• Axial, Sagittal image (약 550 case data)

• 의료진이 annotation tool을이용하여 550 case에대하여 image-level label 생

성

• 의료진이 8 case에대하여 pixel-wise label 생성

• Model architecture

• ResCAM (종양 detection model)

• Input size: 256 by 256

• 50 layer depth (ResNet base)

• Trained using four P100, 16GB Memory

• FCN (종양 segmentation model)

• Input: 256 by 256 (large)

• 16 layer depth (VGGNet base)

• Trained using four P100, 16GB Memory

Experiment setup

20/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Results

21/27

• ResCAM

• Training data: 450 case

• Test data: 100 case

• Training acc.: 87.5%

• Test acc.: 82.5%

• FCN

• Training data: 7 case (226 slices)

• Test data: 1 case (31 slices)

• Pixel-wise accuracy: 98 %

Model training

22/27

Segmentation Results

23/27

Segmentation Results

24/27

• Introduction

• Related work

• Global Average Pooling (GAP)

• Class Activation Map (CAM)

• Fully Convolutional Network (FCN)

• Proposed method

• Experiment setup

• Results

• Conclusion

Conclusion

25/27

• Pixel-wise labeled data를확보하기어려운의료영상 segmentation 문제를

weakly supervised approach로극복

• 대량의 image-level labeled data을학습하여 attention map을추출하는

ResCAM모델적용

• Attention map과소량의 pixel-wise labeled data를학습하여입력영상에

서종양을추출하는 FCN 모델적용

• 개발한 pipeline은다른종류의질병에도적용가능함

• Future work

• Phase 2 (volumetry측정), Phase 3(종양모델생성) 연구진행

Conclusion

26/27

Thank you

27/27