OBSERVATÓR!O2016 [email protected]áticaPuraeAplicada,[email protected]áticaPuraeAplicada,Brazil
This poster focuses on OBSERVATÓR!O2016, a
web-basedplatformforcollecting,structuringandvis-ualizingtheonlineresponsetoRio-2016fromcontentsharedonTwitter.ThisprojectwasdevelopedattheVision and Computer Graphics Laboratory and isbased on two cross related research lines. First, theconception and design of the OBSERVATÓR!O2016website,whichprovidesaspacetoexplorecommentsand images about the Olympics through structuredvisualizations.Second,theongoingdeploymentofre-searchregardingtheapplicationofadigitalmethodtoexplore large setsof images.Thegoal is tohighlighthowweuseDeepLearningapplications-suchasauto-matic image classification - and visualization tech-niques to enhance discoverability and expression ofsubjectfeatureswithinanextensiveimagecollection.
Method: Deep Learning and mosaic visualization
OBSERVATÓR!O2016 collected around 1 milliontweetsfromApril18thtoAugust25th,2016.Inotherto gatherdifferentperspectiveson thedebate abouttheOlympicswecreatedsevenTwittersearchqueries.Thedatawaspresentedineightinteractivevisualiza-tions:
Figure 1. Visualizations
Approximately180,000ofthecollectedtweetsin-cludeduniqueimages.Theproductionoflargesetsof
digitalimagesinauguratesnewavenuesforresearch-ers interested in thehumancreativepractice. In thissense, the investigation of Lev Manovich on digitalmethodstostudyvisualcultureisquiterelevant.Withthis inmind, we explored Rio-2016 images throughDeepLearningapproaches.Astheinvestigationiscur-rentlyongoing,wewillreporttheresearchprocessofasingletask,whichresultedintheTorchMosaicvisu-alization.
During the pre-Olympics, it became evident thatmany of the images gathered by our query scriptswererelatedtotheOlympictorchrelayanddepictedtheiconicobject.Partoftheseimageswereaccompa-niedbytextsthatmentionedthetorch,butnotall.Inaddition,sometweetsmentioningthetorchrelay in-corporated images that didn’t depict the object. Inotherwords,textanalysisalonewasnotsufficienttodetectasetofimagescontainingthetorch.
Thus,wereferredtoaDeepLearningapproachtorecognizetheOlympictorchinourdatabase.Thefieldof visual pattern recognition has been recently im-provedbytheefficientperformanceofConvolutionalNeuralNetworks(CNN).In2012,theworkofKrizhev-skyetal.ontrainingadeepconvolutionalneuralnet-worktoclassifythe1.2millionimagesintheImageNetLSVRC-2010contestinto1000differentclasseshadasubstantialimpactonthecomputervisioncommunity.
Morerecently,andthankstoGoogle,computervi-sion tasks such as image classification have becomemore accessible and applicable. That's because thecompanyreleasedlastyeartheiropensourcesoftwarelibraryTensorFlow.This library runs code for imageclassificationonInception-v3CNNmodel,whichcanberetrainedonadistinct imageclassification task (thisqualityisreferredtoastransferlearning).Bycreatingasetoftrainingimages,itispossibletoupdatethepa-rametersof themodelanduse it torecognizeanewimagecategory.That said,weretrained thenetworkbyshowingitasampleof100manuallylabeledimagescontaining the torch. Finally, the retrained networkran over our database and returned a set of imageswiththeircorrespondingconfidencescoreforthenewcategory.
Figure 2. Confidence score over 83%
Until June 25th, around 1500 images with over85%confidencescorefortheOlympictorchcategoryhadbeenclassifiedbyournetwork.Weusedthemtocreateamosaicvisualizationthatcanbezoomedandpanned.Themosaicideaisthat,givenanimage(targetimage),anotherimage(mosaic)isautomaticallybuildup fromseveral smaller images (tile images).To im-plementthemosaic,weusedaweb-basedviewerforhigh-resolution zoomable images calledOpenSeadragon.
Figure 3. The target image
Figure 4. The mosaic
Figure 5. Tile images
Theorganizationnatureofthemosaicvisualizationis mainly aesthetic. Nevertheless, zooming and pan-ningthemosaicallowstheusertoexploreawidevari-ety of views, and to discover image details and sur-prises suchas the spoofpictureofFofão, aBrazilianfictionalcharacter,carryingthetorch.
ForDH2017postersession,weexpect topresentthe results foranothersubject feature - thesportingdisciplines - and visualisation technique - the vide-osphere -weareworkingonatthemoment.Inaddi-tion,weplantodiscusswithparticipantssomepossi-blescenariosinwhichDeepLearningmodelscouldbeappliedtohelpimagecollectionsbecomemorediscov-erableandexpressive.
Bibliography
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012)."Imagenetclassificationwithdeepconvolutionalneuralnetworks". Advances in neural information processingsystems,pp.1097-1105.
Manovich, L (2012). "How to Compare One Million Im-
ages?”.InBerry,D.(ed),UnderstandingDigitalHumani-ties.Palgrave,pp.249-278.
Szegedy,C.,Liu,W.,Jia,Y.,Sermanet,P.,Reed,S.,Angue-
lov,D.,...&Rabinovich,A.(2015)."Goingdeeperwithconvolutions". Proceedings of the IEEE Conference onComputerVisionandPatternRecognition,pp.1-9.
