computer-aided colorectal tumor classification in nbi endoscopy using cnn features @ fcv2016

Computer-AidedColorectalTumorClassificationinNBIEndoscopyUsing

CNNFeatures

ToruTamaki,ShojiSonoyama,Tsubasa Hirakawa,BisserRaytchev,KazufumiKaneda,Tetsushi Koide,Shigeto Yoshida,HiroshiMieno and

ShinjiTanaka

http://www.sciencekids.co.nz/pictures/humanbody/heartsurfaceanatomy.html

http://sozai.rash.jp/medical/p/000154.htmlhttp://sozai.rash.jp/medical/p/000152.html

http://medical.toykikaku.com/身体のしくみ/腸/http://www.sciencekids.co.nz/pictures/humanbody/humanorgans.html

Rectum

ColoncancerRectalcancer

Colorectalcancer

Colorectal Cancer & Endoscopic Examinationp Over50,000death/yearinJapan

胃

肝臓

肺

乳房

大腸

膵臓

前立腺

0

20000

40000

60000

80000

1970 1980 1990 2000 2010

胃

肝臓

肺

前立腺

大腸

膵臓

乳房（女性）

0

20000

40000

60000

80000

100000

120000

140000

1970 1980 1990 2000 2010

長期的傾向

z 罹患数、死亡数とも胃がんと肝臓がん順位が下がり、肺がんと大腸がん順位が上がっていま

す。

z 罹患数、前立腺がんと女性乳がん増加が顕著です。

【まとめ】 z 2014 年予測値と比べて罹患数が増加した要因、高齢化とがん登録精度向上が考えられま

す。

z 罹患数で大腸、肺、前立腺がんが、死亡数で大腸がんが順位を上げました。

z 罹患、死亡とも胃がんが順位を下げました。

z 大腸、肺がん罹患数増加高齢化（高齢化影響を除くと大腸横い、肺男性で減少）

影響が高いと考えられます。

z 前立腺がん増加、合わせて PSA 検診普及が寄与していると考えられます。

z 胃がん高齢化影響を除くと罹患数・死亡数減少傾向ですが、高齢化影響で増加また横

い（他増加が上回るため順位が下がる）です。

【解釈注意点】 z 予測精度主要部位 5 年間予測について、実測値と比較することで検証されています。

z 予測に誤差が伴います（統計学的なも＋予測不能な要因）。

z 予防危険因子動向、過去罹患率、死亡率変化に表れている部分反映されていると考え

られますが、将来変化など考慮していません。

z 検診動向などによって変動があり得ます（特に前立腺がん罹患）。

＜報道関係からお問い合わせ先＞

国立研究開発法人国立がん研究センター〒104-0045 東京都中央区築地 5-1-1

企画戦略局広報企画室

TEL: 03-3542-2511（代表） FAX: 03-3542-2545 E-mail： ncc-admin @ncc.go.jp

罹患数死亡数

*乳房（女性）データ 2003 年以降

http://www.ncc.go.jp/jp/information/press_release_20150428.html

colorectalliver

pancreas

breast

prostate

stomachlung

Mortality in 2015

NBIsystem

Endoscope Colonsurface

magnification

よつばクリニックブログhttp://yotsuba-clinic.jp/WordPress/?p=63

１分３０秒全大腸内視鏡挿入無痛・水浸法 https://www.youtube.com/watch?v=40L-y9rNOzw

Colorectal Cancer & Endoscopic Examinationp Over50,000death/yearinJapan

EndoscopicExaminationwithNarrowBandImaging(NBI)

p Diagnosebasedontheenhancedmicro-vesselstructure

TypeA

TypeB

TypeC

1

2

3

NBImagnificationfindings

Norm

alAd

vanced

Cancer

胃

肝臓

肺

乳房

大腸

膵臓

前立腺

0

20000

40000

60000

80000

1970 1980 1990 2000 2010

胃

肝臓

肺

前立腺

大腸

膵臓

乳房（女性）

0

20000

40000

60000

80000

100000

120000

140000

1970 1980 1990 2000 2010

長期的傾向

z 罹患数、死亡数とも胃がんと肝臓がん順位が下がり、肺がんと大腸がん順位が上がっていま

す。

z 罹患数、前立腺がんと女性乳がん増加が顕著です。

【まとめ】 z 2014 年予測値と比べて罹患数が増加した要因、高齢化とがん登録精度向上が考えられま

す。

z 罹患数で大腸、肺、前立腺がんが、死亡数で大腸がんが順位を上げました。

z 罹患、死亡とも胃がんが順位を下げました。

z 大腸、肺がん罹患数増加高齢化（高齢化影響を除くと大腸横い、肺男性で減少）

影響が高いと考えられます。

z 前立腺がん増加、合わせて PSA 検診普及が寄与していると考えられます。

z 胃がん高齢化影響を除くと罹患数・死亡数減少傾向ですが、高齢化影響で増加また横

い（他増加が上回るため順位が下がる）です。

【解釈注意点】 z 予測精度主要部位 5 年間予測について、実測値と比較することで検証されています。

z 予測に誤差が伴います（統計学的なも＋予測不能な要因）。

z 予防危険因子動向、過去罹患率、死亡率変化に表れている部分反映されていると考え

られますが、将来変化など考慮していません。

z 検診動向などによって変動があり得ます（特に前立腺がん罹患）。

＜報道関係からお問い合わせ先＞

国立研究開発法人国立がん研究センター〒104-0045 東京都中央区築地 5-1-1

企画戦略局広報企画室

TEL: 03-3542-2511（代表） FAX: 03-3542-2545 E-mail： ncc-admin @ncc.go.jp

罹患数死亡数

*乳房（女性）データ 2003 年以降

http://www.ncc.go.jp/jp/information/press_release_20150428.html

colorectalliver

pancreas

breast

prostate

stomachlung

Mortality in 2015

NBIsystem

Endoscope Colonsurface

magnification

(a) BoVW

(b) VLAD

(c) Fisher vector

Figure 5. Performance with di↵erent number of descriptorsfor encoding.

(a) BoVW (K = 4096)

(b) VLAD (KD = 16, 384)

(c) Fisher vector (2KD = 16, 384)

Figure 6. Computation time di↵erent number of descrip-tors for encoding per image.

REFERENCES

[1] Tamaki, T., Yoshimuta, J., Kawakami, M., Raytchev, B., Kaneda, K., Yoshida, S., Takemura, Y., Onji, K.,Miyaki, R., and Tanaka, S., “Computer-aided colorectal tumor classification in nbi endoscopy using localfeatures,” Medical Image Analysis 17, 78–100 (1 2013).

[2] Csurka, G., Dance, C. R., Fan, L., Willamowski, J., and Bray, C., “Visual categorization with bags of

[Sonoyama etal.,SPEIMI2015]

p Bag-of-Visual-Words,VLAD,FishervectorwithdenseSIFT

p LinearSVMp Recognitionofpatches&video

Type A!

Type B!

Type C!

1

2

3

Microvessels are not observed or extremely opaque.

Fine microvessels are observed around pits, and clear pits can be observed via the nest of microvessels.

Microvessels comprise an irregular network, pits observed via the microvessels are slightly non-distinct, and vessel diameter or distribution is homogeneous. Microvessels comprise an irregular network, pits observed via the microvessels are irregular, and vessel diameter or distribution is heterogeneous. Pits via the microvessels are invisible, irregular vessel diameter is thick, or the vessel distribution is heterogeneous, and a vascular areas are observed.

B, fine microvessels are visible around clearly observed pits (themiddle row of Fig. 5). Type C is divided into three subtypes C1,C2, and C3 according to detailed texture. In type C3, which exhibitsthe most irregular texture, pits are almost invisible because of the

irregularity of tumors, and microvessels are irregular and thick, orheterogeneously distorted (the bottom row of Fig. 5).

This classification has been shown to have a strong correlationwith histological diagnosis (Kanao et al., 2009), as shown in Table 1.

Fig. 4. NBI magnification findings (Kanao et al., 2009).

Fig. 5. Examples of NBI images of types A (top row), B (middle row), and C3 (bottom row).

82 T. Tamaki et al. / Medical Image Analysis 17 (2013) 78–100

[Tamakietal.,MedIA2013]

Theseareshallowmethods.

Recognizing colorectal NBI images

INPUT 32x32

Convolutions SubsamplingConvolutions

C1: feature maps 6@28x28

Subsampling

S2: f. maps6@14x14

S4: f. maps 16@5x5C5: layer120

C3: f. maps 16@10x10

F6: layer 84

Full connectionFull connection

Gaussian connections

OUTPUT 10

• A linear layer with softmax loss as the classifier (pre-dicting the same 1000 classes as the main classifier, butremoved at inference time).

A schematic view of the resulting network is depicted inFigure 3.

6. Training Methodology

GoogLeNet networks were trained using the DistBe-lief [4] distributed machine learning system using mod-est amount of model and data-parallelism. Although weused a CPU based implementation only, a rough estimatesuggests that the GoogLeNet network could be trained toconvergence using few high-end GPUs within a week, themain limitation being the memory usage. Our training usedasynchronous stochastic gradient descent with 0.9 momen-tum [17], fixed learning rate schedule (decreasing the learn-ing rate by 4% every 8 epochs). Polyak averaging [13] wasused to create the final model used at inference time.

Image sampling methods have changed substantiallyover the months leading to the competition, and alreadyconverged models were trained on with other options, some-times in conjunction with changed hyperparameters, suchas dropout and the learning rate. Therefore, it is hard togive a definitive guidance to the most effective single wayto train these networks. To complicate matters further, someof the models were mainly trained on smaller relative crops,others on larger ones, inspired by [8]. Still, one prescrip-tion that was verified to work very well after the competi-tion, includes sampling of various sized patches of the im-age whose size is distributed evenly between 8% and 100%of the image area with aspect ratio constrained to the inter-val [ 34 ,

43 ]. Also, we found that the photometric distortions

of Andrew Howard [8] were useful to combat overfitting tothe imaging conditions of training data.

7. ILSVRC 2014 Classification ChallengeSetup and Results

The ILSVRC 2014 classification challenge involves thetask of classifying the image into one of 1000 leaf-node cat-egories in the Imagenet hierarchy. There are about 1.2 mil-lion images for training, 50,000 for validation and 100,000images for testing. Each image is associated with oneground truth category, and performance is measured basedon the highest scoring classifier predictions. Two num-bers are usually reported: the top-1 accuracy rate, whichcompares the ground truth against the first predicted class,and the top-5 error rate, which compares the ground truthagainst the first 5 predicted classes: an image is deemedcorrectly classified if the ground truth is among the top-5,regardless of its rank in them. The challenge uses the top-5error rate for ranking purposes.

input

Conv7x7+2(S)

MaxPool3x3+2(S)

LocalRespNorm

Conv1x1+1(V)

Conv3x3+1(S)

LocalRespNorm

MaxPool3x3+2(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

MaxPool3x3+2(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

AveragePool5x5+3(V)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

AveragePool5x5+3(V)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

MaxPool3x3+2(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

Conv1x1+1(S)

MaxPool3x3+1(S)

DepthConcat

Conv3x3+1(S)

Conv5x5+1(S)

Conv1x1+1(S)

AveragePool7x7+1(V)

FC

Conv1x1+1(S)

FC

FC

SoftmaxActivation

softmax0

Conv1x1+1(S)

FC

FC

SoftmaxActivation

softmax1

SoftmaxActivation

softmax2

Figure 3: GoogLeNet network with all the bells and whistles.Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilitiesbetween the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-partsat the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, andthe number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096–1000.

neurons in a kernel map). The second convolutional layer takes as input the (response-normalizedand pooled) output of the first convolutional layer and filters it with 256 kernels of size 5⇥ 5⇥ 48.The third, fourth, and fifth convolutional layers are connected to one another without any interveningpooling or normalization layers. The third convolutional layer has 384 kernels of size 3 ⇥ 3 ⇥256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourthconvolutional layer has 384 kernels of size 3 ⇥ 3 ⇥ 192 , and the fifth convolutional layer has 256kernels of size 3⇥ 3⇥ 192. The fully-connected layers have 4096 neurons each.

4 Reducing Overfitting

Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRCmake each training example impose 10 bits of constraint on the mapping from image to label, thisturns out to be insufficient to learn so many parameters without considerable overfitting. Below, wedescribe the two primary ways in which we combat overfitting.

4.1 Data Augmentation

The easiest and most common method to reduce overfitting on image data is to artificially enlargethe dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct formsof data augmentation, both of which allow transformed images to be produced from the originalimages with very little computation, so the transformed images do not need to be stored on disk.In our implementation, the transformed images are generated in Python code on the CPU while theGPU is training on the previous batch of images. So these data augmentation schemes are, in effect,computationally free.

The first form of data augmentation consists of generating image translations and horizontal reflec-tions. We do this by extracting random 224⇥ 224 patches (and their horizontal reflections) from the256⇥256 images and training our network on these extracted patches4. This increases the size of ourtraining set by a factor of 2048, though the resulting training examples are, of course, highly inter-dependent. Without this scheme, our network suffers from substantial overfitting, which would haveforced us to use much smaller networks. At test time, the network makes a prediction by extractingfive 224 ⇥ 224 patches (the four corner patches and the center patch) as well as their horizontalreflections (hence ten patches in all), and averaging the predictions made by the network’s softmaxlayer on the ten patches.

The second form of data augmentation consists of altering the intensities of the RGB channels intraining images. Specifically, we perform PCA on the set of RGB pixel values throughout theImageNet training set. To each training image, we add multiples of the found principal components,

4This is the reason why the input images in Figure 2 are 224⇥ 224⇥ 3-dimensional.

5

Lecun, Y.,Bottou,L.,Bengio,Y.,&Haffner, P.(1998).Gradient-basedlearningappliedtodocumentrecognition.Proceedings oftheIEEE,86(11),2278–2324.doi:10.1109/5.726791

Krizhevsky,A.,Sutskever,I.,&Hinton,G.E.(2012).ImageNet ClassificationwithDeepConvolutionalNeuralNetworks.InF.Pereira,C.J.C.Burges,L.Bottou,&K.Q.Weinberger(Eds.),AdvancesinNeuralInformation ProcessingSystems25(pp.1097–1105).CurranAssociates,Inc.Retrievedfromhttp://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

Szegedy,C.,Liu,W.,Jia,Y.,Sermanet,P.,Reed,S.,Anguelov,D.,…Rabinovich, A.(2015).GoingDeeperWithConvolutions.InTheIEEEConference on ComputerVisionandPatternRecognition(CVPR).

Deep Learning on going

Deep learning >> shallow learning

Deeper the betterResultinILSVRCovertheyears

1.9x 2.8x

1.4x

Revolution of Depth11x11 conv, 96, /4, pool/2

5x5 conv, 256, pool/2

3x3 conv, 384

3x3 conv, 384


fc, 4096

fc, 4096

fc, 1000

AlexNet, 8 layers(ILSVRC 2012)

3x3 conv, 64


3x3 conv, 128


3x3 conv, 256

3x3 conv, 256

3x3 conv, 256


3x3 conv, 512

3x3 conv, 512

3x3 conv, 512


3x3 conv, 512

3x3 conv, 512

3x3 conv, 512


fc, 4096

fc, 4096

fc, 1000

VGG, 19 layers(ILSVRC 2014)

input

Conv7x7+ 2(S)

MaxPool 3x3+ 2(S)

LocalRespNorm

Conv1x1+ 1(V)

Conv3x3+ 1(S)

LocalRespNorm

MaxPool 3x3+ 2(S)

Conv Conv Conv Conv1x1+ 1(S) 3x3+ 1(S) 5x5+ 1(S) 1x1+ 1(S)

Conv Conv MaxPool 1x1+ 1(S) 1x1+ 1(S) 3x3+ 1(S)

Dept hConcat



Dept hConcat

MaxPool 3x3+ 2(S)



Dept hConcat



AveragePool 5x5+ 3(V)

Dept hConcat



Dept hConcat



Dept hConcat




Dept hConcat

MaxPool 3x3+ 2(S)



Dept hConcat



Dept hConcat


FC

Conv1x1+ 1(S)

FC

FC

Soft maxAct ivat ion

soft max0

Conv1x1+ 1(S)

FC

FC


soft max1


soft max2

GoogleNet, 22 layers(ILSVRC 2014)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

Slide from: ILSVR2015 workshop

AlexNet, 8 layers(ILSVRC 2012)

Revolution of DepthResNet, 152 layers

(ILSVRC 2015)

3x3 conv, 64


3x3 conv, 128


3x3 conv, 256

3x3 conv, 256

3x3 conv, 256


3x3 conv, 512

3x3 conv, 512

3x3 conv, 512


3x3 conv, 512

3x3 conv, 512

3x3 conv, 512


fc, 4096

fc, 4096

fc, 1000

11x11 conv, 96, /4, pool/2


3x3 conv, 384

3x3 conv, 384


fc, 4096

fc, 4096

fc, 1000

1x1 conv, 64

3x3 conv, 64

1x1 conv, 256

1x1 conv, 64

3x3 conv, 64

1x1 conv, 256

1x1 conv, 64

3x3 conv, 64

1x1 conv, 256

1x2 conv, 128, /2

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 128

3x3 conv, 128

1x1 conv, 512

1x1 conv, 256, /2

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 256

3x3 conv, 256

1x1 conv, 1024

1x1 conv, 512, /2

3x3 conv, 512

1x1 conv, 2048

1x1 conv, 512

3x3 conv, 512

1x1 conv, 2048

1x1 conv, 512

3x3 conv, 512

1x1 conv, 2048

ave pool, fc 1000

7x7 conv, 64, /2, pool/2

VGG, 19 layers(ILSVRC 2014)

Kaiming He, Xiangyu Zhang, Shaoqing Ren, & Jian Sun. “Deep Residual Learning for Image Recognition”. arXiv 2015.

p Networksbecomedeeperyearbyyearfor:p Scenerecognitionp Categoryrecognitionp Andmanyothertasks

p But…p Howdeep?p Whatarchitecture?p Reallynecessary?

Slidefrom:Kaiming He:Deepresiduallearning,ILSVRC2015@ICCV2015

Using the off-the-shelf features

SharifRazavian,A.,Azizpour,H.,Sullivan,J.,&Carlsson,S.(2014).CNNFeaturesOff-the-Shelf:AnAstoundingBaselineforRecognition. InTheIEEEConference onComputerVisionandPatternRecognition(CVPR)Workshops.

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilitiesbetween the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-partsat the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, andthe number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096–1000.









5

Deep learning in general and deep convolutional neural networks (CNN) [3] [4] in particular, is a composite model of neural networks. Deep learning enjoys good success since 2006, and is shown to achieve improvement in classifying images, audio, and speech data. All deep learning models require a substantial amount of training instances to avoid the problem of over-fitting. Some research works in the medical field have started employing a deep architecture [11] [12]. In this work, we do not use deep learning to directly training an OM classifier. Instead, we use the unsupervised layers (unsupervised with respect to OM) of CNN to learn the structures of non-OM-related images, and then use that learned representation to model OM data. In other words, we use CNN to perform transfer representation learning.

Transfer learning, defined as the ability of a system to recognize and apply knowledge and skills learned in previous tasks to a novel task, is on the rise in recent years and has been applied in several data mining tasks, e.g., [2] [8] [9] [10]. Our work transfers representations learned from one domain to detect OM, and we refer this approach as transfer representation learning.

III. METHODS

As depicted in the introductory section, our scheme is composed of four steps. This section details these steps.

We start with unsupervised codebook construction. On a large ImageNet dataset, we learn the representation of these images using a variant of deep CNN, Alexnet [4]. Alexnet, the winner of the world-wide image recognition competition (ILSVRC) in 2012, contains eight neural network layers. The first five are convolutional and the remaining three are fully-connected. Different layers represent different levels of abstraction concepts. We utilize Alexnet in Caffe [5] as our foundation to build our encoder to capture generic visual features.

For each image input, we obtain a feature vector using the codebook. The information of the image moves from the input layer to the output layer through the inner layers. Each layer is a weighted combination of the previous layer and stands for a feature representation of the input image. Since the computation is hierarchical, higher layers intuitively represent higher concepts. For images, the neurons from lower levels describe rudimental perceptual elements like edges and corners, while higher layers represent object parts such as contours and categories. To capture higher-level abstractions, we extract transfer-learned features of OM images from the fifth, sixth and seventh layer, denoted as pool5, fc6 and fc7 in Fig.1, respectively , , 3 , 3 ,features from the eighth layer is that it only produces the probability of the class prediction, and it is not a representation of the input image.

Once we have transfer-learned feature vectors of the 1,195 collected OM images, we perform supervised learning by training a Support Vector Machine (SVM) classifier [14]. We choose SVM to be our model since it is an effective classifier widely used by prior work. As usual, we scale features to the same range and find parameters through cross validation. For comparing to the previous work fairly, we select radial basis function (RBF) kernel.

Figure 1. The flowchart of our transfer representation learning algorithm

(otitis media photo is from [17])

To further improve classification accuracy, we experiment with two feature fusion schemes, which combine features hand-crafted by human heuristics in [7] with features learned from our codebook. In the first scheme, we combine transfer-learned and hand-crafted OM features to form fusion feature vectors. We then deploy the supervised learning upon the fused feature vectors to train a SVM classifier. In the second scheme, we use a two-layer classifier fusion structure mentioned in [7]. Concisely, in the first layer, we train different classifiers upon different feature sets separately. We then combine all the outputs from the first layer to train the classifier in the second layer.

Fig. 2 summarizes our transfer representation learning approaches. On the top of the figure depict two feature-learning schemes: the transfer-learned on the left-hand side and the hand-crafted on the right. The solid lines depict how OM features are extracted via the transfer-learned codebook, whereas the dash-dot lines represent the flow of hand-crafted OM feature extraction. The bottom half of the figure describes two fusion schemes. Whereas the dashed lines illustrate the feature fusion by concatenating two feature sets, the dotted lines show the second fusion scheme at the classifier level. At the bottom of the figure, the four classification flows yield their respective OM-prediction decision. From the left-hand side to the right-hand side, they are transfer-learned features only, feature-level fusion, classifier-level fusion, and hand-crafted features only.

IV. EXPERIMENTAL RESULTS AND DISCUSSION

We conducted two sets of experiments. Subsection �-A

reports OM classification performance by using our proposed transfer representation learning approach. The effect of transfer representation learning on improving OM

classification is evaluated in Subsection �-B.

A. Results of Transfer Representation Learning

Table 1 compares OM classification results for different feature representations. All experiments are conducted by using 10-fold SVM classification. The measures of results reflect the discrimination capability of the features.

The first two rows in Table 1 show the results of human-heuristic methods (hand-crafted), followed by our proposed transfer-learned approach. The eardrum segmentation, denoted as seg , identifies the eardrum by

712

Shie,C.-K.,Chuang, C.-H.,Chou, C.-N.,Wu,M.-H.,&Chang,E.Y.(2015).TransferRepresentationLearningforMedicalImageAnalysis.InThe37thAnnualInternationalConference oftheIEEEEngineeringinMedicineandBiologySociety(EMBC) (pp. 711–714).

Convolution

Input Li

Fully connected

1st fullypropagatedactivationlayer

L2 L5(Decafs) L6 (Decaf6) L7 (Decafi)9216 4096 4096

Activations Activations Activations

Pre -LearnedCNN (Decaf)

High. Rep. Of Low -Levelfeatures (PiCoDes)

IIMedical Dataset 11/

/1!Feature Extraction +

Fusion (optional)

[x1 xn]

Standardization

[y1, ,yn] 1

Individual PathologyDetection

(Binary ClassificationUsing Linear SVM)

Low -Level:GistBoWLBP

Proc. of SPIE Vol. 9414 94140V-3

Downloaded From: http://spiedigitallibrary.org/ on 10/22/2015 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

Bar,Y.,Diamant,I.,Wolf,L.,&Greenspan,H.(2015).Deeplearningwithnon-medicaltrainingusedforchestpathologyidentification.InL.M.Hadjiiski &G.D.Tourassi (Eds.),SPIEMedicalImaging(p.94140V).InternationalSocietyforOpticsandPhotonics.doi:10.1117/12.2083124

1. Pre-trained CNN with imageNet

2. Use a layer

3. As a feature

Success in medical image analysis

DeCAF

AlexNet

AlexNet (OverFeat)

Using the off-the-shelf features

SharifRazavian,A.,Azizpour,H.,Sullivan,J.,&Carlsson,S.(2014).CNNFeaturesOff-the-Shelf:AnAstoundingBaselineforRecognition. InTheIEEEConference onComputerVisionandPatternRecognition(CVPR)Workshops.










5

Deep learning in general and deep convolutional neural networks (CNN) [3] [4] in particular, is a composite model of neural networks. Deep learning enjoys good success since 2006, and is shown to achieve improvement in classifying images, audio, and speech data. All deep learning models require a substantial amount of training instances to avoid the problem of over-fitting. Some research works in the medical field have started employing a deep architecture [11] [12]. In this work, we do not use deep learning to directly training an OM classifier. Instead, we use the unsupervised layers (unsupervised with respect to OM) of CNN to learn the structures of non-OM-related images, and then use that learned representation to model OM data. In other words, we use CNN to perform transfer representation learning.

Transfer learning, defined as the ability of a system to recognize and apply knowledge and skills learned in previous tasks to a novel task, is on the rise in recent years and has been applied in several data mining tasks, e.g., [2] [8] [9] [10]. Our work transfers representations learned from one domain to detect OM, and we refer this approach as transfer representation learning.

III. METHODS

As depicted in the introductory section, our scheme is composed of four steps. This section details these steps.

We start with unsupervised codebook construction. On a large ImageNet dataset, we learn the representation of these images using a variant of deep CNN, Alexnet [4]. Alexnet, the winner of the world-wide image recognition competition (ILSVRC) in 2012, contains eight neural network layers. The first five are convolutional and the remaining three are fully-connected. Different layers represent different levels of abstraction concepts. We utilize Alexnet in Caffe [5] as our foundation to build our encoder to capture generic visual features.

For each image input, we obtain a feature vector using the codebook. The information of the image moves from the input layer to the output layer through the inner layers. Each layer is a weighted combination of the previous layer and stands for a feature representation of the input image. Since the computation is hierarchical, higher layers intuitively represent higher concepts. For images, the neurons from lower levels describe rudimental perceptual elements like edges and corners, while higher layers represent object parts such as contours and categories. To capture higher-level abstractions, we extract transfer-learned features of OM images from the fifth, sixth and seventh layer, denoted as pool5, fc6 and fc7 in Fig.1, respectively , , 3 , 3 ,features from the eighth layer is that it only produces the probability of the class prediction, and it is not a representation of the input image.

Once we have transfer-learned feature vectors of the 1,195 collected OM images, we perform supervised learning by training a Support Vector Machine (SVM) classifier [14]. We choose SVM to be our model since it is an effective classifier widely used by prior work. As usual, we scale features to the same range and find parameters through cross validation. For comparing to the previous work fairly, we select radial basis function (RBF) kernel.

Figure 1. The flowchart of our transfer representation learning algorithm

(otitis media photo is from [17])

To further improve classification accuracy, we experiment with two feature fusion schemes, which combine features hand-crafted by human heuristics in [7] with features learned from our codebook. In the first scheme, we combine transfer-learned and hand-crafted OM features to form fusion feature vectors. We then deploy the supervised learning upon the fused feature vectors to train a SVM classifier. In the second scheme, we use a two-layer classifier fusion structure mentioned in [7]. Concisely, in the first layer, we train different classifiers upon different feature sets separately. We then combine all the outputs from the first layer to train the classifier in the second layer.

Fig. 2 summarizes our transfer representation learning approaches. On the top of the figure depict two feature-learning schemes: the transfer-learned on the left-hand side and the hand-crafted on the right. The solid lines depict how OM features are extracted via the transfer-learned codebook, whereas the dash-dot lines represent the flow of hand-crafted OM feature extraction. The bottom half of the figure describes two fusion schemes. Whereas the dashed lines illustrate the feature fusion by concatenating two feature sets, the dotted lines show the second fusion scheme at the classifier level. At the bottom of the figure, the four classification flows yield their respective OM-prediction decision. From the left-hand side to the right-hand side, they are transfer-learned features only, feature-level fusion, classifier-level fusion, and hand-crafted features only.

IV. EXPERIMENTAL RESULTS AND DISCUSSION

We conducted two sets of experiments. Subsection �-A

reports OM classification performance by using our proposed transfer representation learning approach. The effect of transfer representation learning on improving OM

classification is evaluated in Subsection �-B.

A. Results of Transfer Representation Learning

Table 1 compares OM classification results for different feature representations. All experiments are conducted by using 10-fold SVM classification. The measures of results reflect the discrimination capability of the features.

The first two rows in Table 1 show the results of human-heuristic methods (hand-crafted), followed by our proposed transfer-learned approach. The eardrum segmentation, denoted as seg , identifies the eardrum by

712

Shie,C.-K.,Chuang, C.-H.,Chou, C.-N.,Wu,M.-H.,&Chang,E.Y.(2015).TransferRepresentationLearningforMedicalImageAnalysis.InThe37thAnnualInternationalConference oftheIEEEEngineeringinMedicineandBiologySociety(EMBC) (pp. 711–714).

Convolution

Input Li

Fully connected

1st fullypropagatedactivationlayer

L2 L5(Decafs) L6 (Decaf6) L7 (Decafi)9216 4096 4096

Activations Activations Activations

Pre -LearnedCNN (Decaf)

High. Rep. Of Low -Levelfeatures (PiCoDes)

IIMedical Dataset 11/

/1!Feature Extraction +

Fusion (optional)

[x1 xn]

Standardization

[y1, ,yn] 1

Individual PathologyDetection

(Binary ClassificationUsing Linear SVM)

Low -Level:GistBoWLBP

Proc. of SPIE Vol. 9414 94140V-3

Downloaded From: http://spiedigitallibrary.org/ on 10/22/2015 Terms of Use: http://spiedigitallibrary.org/ss/TermsOfUse.aspx

Bar,Y.,Diamant,I.,Wolf,L.,&Greenspan,H.(2015).Deeplearningwithnon-medicaltrainingusedforchestpathologyidentification.InL.M.Hadjiiski &G.D.Tourassi (Eds.),SPIEMedicalImaging(p.94140V).InternationalSocietyforOpticsandPhotonics.doi:10.1117/12.2083124

1. Pre-trained CNN with imageNet

2. Use a layer

3. As a feature

Success in medical image analysis

DeCAF

AlexNet

AlexNet (OverFeat)

FCV2016P2-8:ObjectRecognitionUsingSizeInformationBasedonPre-trainedConvolutionalNeuralNetwork

Our work: compare all layers










5

linear SVM(as is)

p DifferentLayersp Convolutionp Normalizationp Poolingp Fullconnect

p ThreeCNNsp AlexNetp CaffeNetp GoogLeNet

p Caffe

•A

linearlayerwith

softmax

lossas

theclassifier(pre-

dictingthe

same

1000classesasthe

main

classifier,butrem

ovedatinference

time).

Aschem

aticview

oftheresulting

network

isdepicted

inFigure

3.

6.TrainingM

ethodology

GoogLeN

etnetw

orksw

eretrained

usingthe

DistB

e-lief

[4]distributed

machine

learningsystem

usingm

od-est

amount

ofm

odeland

data-parallelism.

Although

we

useda

CPU

basedim

plementation

only,arough

estimate

suggeststhat

theG

oogLeNet

network

couldbe

trainedto

convergenceusing

fewhigh-end

GPU

sw

ithina

week,the

main

limitation

beingthe

mem

oryusage.O

urtrainingused

asynchronousstochastic

gradientdescentwith

0.9m

omen-

tum[17],fixed

learningrate

schedule(decreasing

thelearn-

ingrate

by4%

every8

epochs).Polyakaveraging

[13]was

usedto

createthe

finalmodelused

atinferencetim

e.Im

agesam

plingm

ethodshave

changedsubstantially

overthe

months

leadingto

thecom

petition,and

alreadyconverged

modelsw

eretrained

onw

ithotheroptions,som

e-tim

esin

conjunctionw

ithchanged

hyperparameters,

suchas

dropoutand

thelearning

rate.Therefore,

itis

hardto

givea

definitiveguidance

tothe

mosteffective

singlew

ayto

trainthese

networks.To

complicate

mattersfurther,som

eofthe

modelsw

erem

ainlytrained

onsm

allerrelativecrops,

otherson

largerones,inspired

by[8].

Still,oneprescrip-

tionthatw

asverified

tow

orkvery

wellafter

thecom

peti-tion,includes

sampling

ofvarioussized

patchesofthe

im-

agew

hosesize

isdistributed

evenlybetw

een8%

and100%

oftheim

agearea

with

aspectratioconstrained

tothe

inter-val

[34 ,

43 ].A

lso,we

foundthatthe

photometric

distortionsofA

ndrewH

oward

[8]were

usefultocom

batoverfittingto

theim

agingconditions

oftrainingdata.

7.IL

SVR

C2014

Classification

Challenge

Setupand

Results

TheILSV

RC

2014classification

challengeinvolves

thetask

ofclassifyingthe

image

intoone

of1000leaf-node

cat-egories

inthe

Imagenethierarchy.There

areabout1.2

mil-

lionim

agesfortraining,50,000

forvalidationand

100,000im

agesfor

testing.Each

image

isassociated

with

oneground

truthcategory,and

performance

ism

easuredbased

onthe

highestscoring

classifierpredictions.

Two

num-

bersare

usuallyreported:

thetop-1

accuracyrate,

which

compares

theground

truthagainstthe

firstpredictedclass,

andthe

top-5error

rate,which

compares

theground

truthagainst

thefirst

5predicted

classes:an

image

isdeem

edcorrectly

classifiedif

theground

truthis

among

thetop-5,

regardlessofits

rankin

them.The

challengeuses

thetop-5

errorrateforranking

purposes.

input

Conv7x7+

2(S)

MaxPool

3x3+2(S)

LocalRespNorm

Conv1x1+

1(V)

Conv3x3+

1(S)

LocalRespNorm

MaxPool

3x3+2(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

MaxPool

3x3+2(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

AveragePool5x5+

3(V)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

AveragePool5x5+

3(V)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

MaxPool

3x3+2(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)

Conv1x1+

1(S)Conv

1x1+1(S)

MaxPool

3x3+1(S)

DepthConcat

Conv3x3+

1(S)Conv

5x5+1(S)

Conv1x1+

1(S)

AveragePool7x7+

1(V)

FC

Conv1x1+

1(S)

FC FC

SoftmaxActivation

softmax0

Conv1x1+

1(S)

FC FC

SoftmaxActivation

softmax1

SoftmaxActivation

softmax2

Figure3:G

oogLeNetnetw

orkw

ithallthe

bellsand

whistles.

AlexNet (Cafe)

CaffeNet(Cafe)

GoogLeNet (Cafe)

Experimental settingTraining• 907 NBI image patches

(Type A: 359, Type B: 461, Type C3: 87)trimmed by medical doctors and endoscopists.

• Different sizes: resized to a fixed size (227x227)• Linear SVM (with parameter C by 3-fold CV)• Average accuracy of 10-fold CV

Type A:

Type B:

Type C3:

Image patches

trimming

patch

Full size frame

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

�

��

��

��

��

�

��

��

��

��

��

��

��

��

1st layer 2nd layer

Visualization by draw_net.py in Caffe

Layers and features: an example










5

Results for AlexNet��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��










5

dimension

accuracy

Results for AlexNet��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��










5

pool2: 95.8±2.6% fc6: 95.0±2.9%

43264

4096

Results for CaffeNet��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��










5

pool then normalize

conv3: 96.8±2.0%

64896

fc6: 95.8±2.6%

4096










5

Results for CaffeNet��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Combinationsdon’t help

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Results for GoogLeNetpool3/3x3 s2 : 96.9±1.3%

94080

Results for GoogLeNet

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Type A!

Type B!

Type C!

1 2 3

Microvessels are not observed or extrem

ely opaque.

Fine microvessels are observed around pits, and clear pits

can be observed via the nest of microvessels.

Microvessels com

prise an irregular network, pits observed

via the microvessels are slightly non-distinct, and vessel

diameter or distribution is hom

ogeneous. M

icrovessels comprise an irregular netw

ork, pits observed via the m

icrovessels are irregular, and vessel diameter or

distribution is heterogeneous. Pits via the m

icrovessels are invisible, irregular vessel diam

eter is thick, or the vessel distribution is heterogeneous, and a vascular areas are observed.

Fig.1.N

BI

endoscopicim

agesand

medical

categories[18].

Fig.2.Size

distributionof

image

patchesfor

eachcategory

ofN

BI.

CN

Nm

odels,incontrastto

existingw

ork[14]–[16]

thatusesonly

fewlayer

features.Discussions

basedon

obtainedresults

isanother

contributionof

thispaper

inorder

todesign

andpropose

abetter

CN

Nm

odelfor

thiskind

ofspecific

task:N

BI

colorectalpolyp

image

recognition.

III.M

ET

HO

D

Thetarget

datasetfor

thecom

parisonin

thispaper

isthe

NB

Iim

agedataset

including908

NB

Ipatches

collectedfrom

endoscopicexam

inationsat

Hiroshim

aU

niversity1,

andlabeled

basedon

theN

BI

magnification

findings[18]–[20]

which

categorizesappearances

oftum

orsinto

typeA

,B

,and

C,

andtype

Cis

furthersub-classified

intoC

1,C

2,and

C3

basedon

microvessel

structures(see

Figure1).

Inthis

work

we

useonly

typesA

,B,and

C3

byfollow

ingprevious

work

[12],[13],[21],[22].Apatch

istrim

med

froma

largerfram

eof

theentire

endoscopicim

age,so

thatthe

trimm

edrectangle

regionw

ellrepresents

thetypical

texturepattern

ofthe

colorectalpolypappearing

inthe

frame.D

ifferentpatcheshas

thereforedifferentsizes

rangingbetw

eenabout150

to600

pixels.Distributions

ofsizes

foreach

ofthree

NB

Icategories

areshow

nin

Figure2.

Therem

ightbe

differentw

aysto

encodethese

image

patchesw

ithC

NN

features.For

example,

Krizhevsky

etal.

[23]crop

anim

ageinto

severalpatches

offixed

sizefor

dataaugm

entation,then

takethe

averageof

predictionsof

thosepatches.R

azavianet

al.[14]em

ploya

similar

way

butm

oreaggressively

inorder

toobtain

more

thanone

hundredpatches

froma

singletest

image.

Another

choicem

ightbe

1Thestudy

was

conductedw

ithan

approvalfromthe

Hiroshim

aU

niversityH

ospitalethics

comm

ittee,and

aninform

edconsent

was

obtainedfrom

thepatients

and/orfam

ilym

embers

forthe

endoscopicexam

ination.

��

��

��

��

��

��

��

��

��

��

Fig.3.C

NN

sused

forexperim

ents:(left)

AlexN

et,(middle)

Caffe

Net,and

(right)GoogLeN

et.Gray

polygons(and

white

ellipses)representdata,orCN

Nfeatures,

andcolor

rectanglesrepresent

dataprocessing

ineach

layer.N

otethat

numbers

oflayers

arecounted

differentlyin

differentliterature.

Here

alayer

isdata

aftera

processing(such

asconvolution,pooling,norm

alization,etc.).N

umbers

innam

esof

layersshow

apack

ofthese

layers(for

example,

conv5and

pool5are

5thlayer

ofthe

CN

Nm

odelin

theliterature).

pooling

convolution

full connect

full connect

Full connectbranch Full connect

Full connectbranch

Results for hand-crafted features

BoVW Fisher vector VLAD

97.2±0.3%

32768

Discussions

AlexNet CaffeNet GoogLeNet

p Firstfewlayersareenoughtoreachmaximumaccuracy.

p Shallowisenough.p BoVW,FV,VLADworkwell.

Discussions


p Poolinglayersmayworkwell.

Discussions


p Deeperlayersdonothelpsomuch.p Fullconnectedlayerskeepperformancewhilereducingdimension.

p Dimensionalityreduction(PCA,etc.)mayreplaceit.

Discussions


p Dataitself(pixelintensity)isnouse.p Probabilitycanbeafeature?!

59.0±4.3%

85.4±4.7% 85.7±4.7%

80.7±2.4%

Conclusions

• Comparedoff-the-shelffeaturesfromthreeCNNmodels

• ShallowisenoughforNBIimageclassification• Apossiblearchitectureis:










5

Dimensionreduction(PCA)

Classifier(SVM)

ShallowCNNwith

fewerfilters

Results for GoogLeNet

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Results for GoogLeNet: pooling

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

Results for GoogLeNet: full connect

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

��

computer-aided colorectal tumor classification in nbi endoscopy using cnn features @ fcv2016

Technology