OBSERVATÓR!O2016 JuliaGianellajulia@impa.brInstitutodeMatemáticaPuraeAplicada,BrazilLuizVelholvelho@impa.brInstitutodeMatemáticaPuraeAplicada,Brazil

This poster focuses on OBSERVATÓR!O2016, a

web-basedplatformforcollecting,structuringandvis-ualizingtheonlineresponsetoRio-2016fromcontentsharedonTwitter.ThisprojectwasdevelopedattheVision and Computer Graphics Laboratory and isbased on two cross related research lines. First, theconception and design of the OBSERVATÓR!O2016website,whichprovidesaspacetoexplorecommentsand images about the Olympics through structuredvisualizations.Second,theongoingdeploymentofre-searchregardingtheapplicationofadigitalmethodtoexplore large setsof images.Thegoal is tohighlighthowweuseDeepLearningapplications-suchasauto-matic image classification - and visualization tech-niques to enhance discoverability and expression ofsubjectfeatureswithinanextensiveimagecollection.

Method: Deep Learning and mosaic visualization

OBSERVATÓR!O2016 collected around 1 milliontweetsfromApril18thtoAugust25th,2016.Inotherto gatherdifferentperspectiveson thedebate abouttheOlympicswecreatedsevenTwittersearchqueries.Thedatawaspresentedineightinteractivevisualiza-tions:

Figure 1. Visualizations

Approximately180,000ofthecollectedtweetsin-cludeduniqueimages.Theproductionoflargesetsof

digitalimagesinauguratesnewavenuesforresearch-ers interested in thehumancreativepractice. In thissense, the investigation of Lev Manovich on digitalmethodstostudyvisualcultureisquiterelevant.Withthis inmind, we explored Rio-2016 images throughDeepLearningapproaches.Astheinvestigationiscur-rentlyongoing,wewillreporttheresearchprocessofasingletask,whichresultedintheTorchMosaicvisu-alization.

During the pre-Olympics, it became evident thatmany of the images gathered by our query scriptswererelatedtotheOlympictorchrelayanddepictedtheiconicobject.Partoftheseimageswereaccompa-niedbytextsthatmentionedthetorch,butnotall.Inaddition,sometweetsmentioningthetorchrelay in-corporated images that didn’t depict the object. Inotherwords,textanalysisalonewasnotsufficienttodetectasetofimagescontainingthetorch.

Thus,wereferredtoaDeepLearningapproachtorecognizetheOlympictorchinourdatabase.Thefieldof visual pattern recognition has been recently im-provedbytheefficientperformanceofConvolutionalNeuralNetworks(CNN).In2012,theworkofKrizhev-skyetal.ontrainingadeepconvolutionalneuralnet-worktoclassifythe1.2millionimagesintheImageNetLSVRC-2010contestinto1000differentclasseshadasubstantialimpactonthecomputervisioncommunity.

Morerecently,andthankstoGoogle,computervi-sion tasks such as image classification have becomemore accessible and applicable. That's because thecompanyreleasedlastyeartheiropensourcesoftwarelibraryTensorFlow.This library runs code for imageclassificationonInception-v3CNNmodel,whichcanberetrainedonadistinct imageclassification task (thisqualityisreferredtoastransferlearning).Bycreatingasetoftrainingimages,itispossibletoupdatethepa-rametersof themodelanduse it torecognizeanewimagecategory.That said,weretrained thenetworkbyshowingitasampleof100manuallylabeledimagescontaining the torch. Finally, the retrained networkran over our database and returned a set of imageswiththeircorrespondingconfidencescoreforthenewcategory.

Figure 2. Confidence score over 83%

Until June 25th, around 1500 images with over85%confidencescorefortheOlympictorchcategoryhadbeenclassifiedbyournetwork.Weusedthemtocreateamosaicvisualizationthatcanbezoomedandpanned.Themosaicideaisthat,givenanimage(targetimage),anotherimage(mosaic)isautomaticallybuildup fromseveral smaller images (tile images).To im-plementthemosaic,weusedaweb-basedviewerforhigh-resolution zoomable images calledOpenSeadragon.

Figure 3. The target image

Figure 4. The mosaic

Figure 5. Tile images

Theorganizationnatureofthemosaicvisualizationis mainly aesthetic. Nevertheless, zooming and pan-ningthemosaicallowstheusertoexploreawidevari-ety of views, and to discover image details and sur-prises suchas the spoofpictureofFofão, aBrazilianfictionalcharacter,carryingthetorch.

ForDH2017postersession,weexpect topresentthe results foranothersubject feature - thesportingdisciplines - and visualisation technique - the vide-osphere -weareworkingonatthemoment.Inaddi-tion,weplantodiscusswithparticipantssomepossi-blescenariosinwhichDeepLearningmodelscouldbeappliedtohelpimagecollectionsbecomemorediscov-erableandexpressive.

Bibliography

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012)."Imagenetclassificationwithdeepconvolutionalneuralnetworks". Advances in neural information processingsystems,pp.1097-1105.

Manovich, L (2012). "How to Compare One Million Im-

ages?”.InBerry,D.(ed),UnderstandingDigitalHumani-ties.Palgrave,pp.249-278.

Szegedy,C.,Liu,W.,Jia,Y.,Sermanet,P.,Reed,S.,Angue-

lov,D.,...&Rabinovich,A.(2015)."Goingdeeperwithconvolutions". Proceedings of the IEEE Conference onComputerVisionandPatternRecognition,pp.1-9.