sravanthi kota venkata, ikkjin ahn, donghwan jeon ...cseweb.ucsd.edu/~mbtaylor/vision/sd-vbs.pdf ·...
TRANSCRIPT
SravanthiKotaVenkata,IkkjinAhn,DonghwanJeon,AnshumanGupta,ChrisLouie,SaturninoGarcia,SergeBelongie,
MichaelB.Taylor
ComputerScienceandEngineeringUniversityofCalifornia,SanDiego
hIp://parallel.ucsd.edu/vision
VisionisanexciMngapplicaMondomainformany‐coreandmulM‐coresystems
• “Enablingcomputerstosee”willhaveatangibleandimmediateimpactonpeople’slives
• LimitlessthirstforcomputaMon– Largerimagesizes– Moreaccurateanalyses– Matchandsearchagainstlargerdatabases– Runthealgorithminreal‐Mme,orevensuper‐real‐Mme
• Fullofparallelismtoputthoseidlecorestowork– i.e.driveuserdemandforfuturemulM‐coreprocessorsandkeep
Moore’sLawgoing
• Enormousprogressincomputervisionresearchoverthelastdecade,andmoretocome– Someproblemsarenowevenconsidered“solved.”
AFewExamplesofVision’sCurrentandPotenMalImpactonOurLives
• Auto‐focuscamerasandcell‐phonesthroughfacedetecMon
• HelpdoctorsperformsurgeryonpaMentsthousandsofmilesaway
• Allowplanestoflythemselves(e.g.,UAVs)
• Enable“carsthatcan’tcrash”
• EnablemachinesthatautomaMcallyeducateourkids(!)
SD‐VBS:TheSanDiegoVisionBenchmarkSuite• Aim:MakevisionmoreaccessibletomulM‐coreresearchers
throughacomprehensiveandeasytousesuite.
• 9benchmarksin4differentareas.
Benchmark Concentra.onArea
DisparityMap MoMon,TrackingandStereoVision
FeatureTracking MoMon,TrackingandStereoVision
ImageSegmentaMon ImageAnalysis
ScaleInvariantFeatureTransform(SIFT) ImageAnalysis
RobotLocalizaMon ImageUnderstanding
SupportVectorMachines(SVM) ImageUnderstanding
FaceDetecMon ImageUnderstanding
ImageSMtch ImageProcessingandFormaMon
TextureSynthesis ImageProcessingandFormaMon
DesigngoalsofSD‐VBS
• CleanimplementaMonsforeaseofanalysisandtransformaMonbyresearchersortheirtools
– “Clean”CandMATLABversions
– Nodependenceoncustomlibraries
• ComprehensivecollecMonofrepresentaMvebenchmarks
• LowcomplexityofporMngandparallelizingtodifferentpla_orms
SD‐VBS:9applicaMons,>25kernels
Sort
SSD
CorrelaMon
AdjacencyMatrix
FilterBanks
Eigensolver
QRFactorizaMon
ParMcleFilter
PhysicalModel
PCA
Sampling
ConjugateMatrix
MatrixOps
AffineTransforms
MatrixInvert
AreaSum
Gradient
FeatureTracking
DisparityMap
LSSolver
SVD
ANMS
GaussianFilter
ImageS.tch
RectFilter
Adaboost
IntegralImage
FaceDetec.on
ImageSegmenta.on
RobotLocaliza.on
TextureSynthesis
SupportVector
Machines
SIFT
InterpolaMon
RichReuseofKernelsinVision
Sort
SSD
CorrelaMon
AdjacencyMatrix
FilterBanks
Eigensolver
QRFactorizaMon
ParMcleFilter
PhysicalModel
PCA
Sampling
ConjugateMatrix
MatrixOps
AffineTransforms
MatrixInvert
AreaSum
Gradient
FeatureTracking
DisparityMap
LSSolver
SVD
ANMS
GaussianFilter
ImageS.tch
RectFilter
Adaboost
IntegralImage
FaceDetec.on
ImageSegmenta.on
RobotLocaliza.on
TextureSynthesis
SupportVector
Machines
SIFT
InterpolaMon
OverviewofTalk
• IntroducMontoSD‐VBS• VisionBenchmarksOverview
– foreach{FeatureTracking,Disparity,ImageSMtch,SIFT,SegmentaMon,SVM,RobotLocalizaMon,
TextureSynthesis}:BriefDemo
AlgorithmDescripMonAnalysisofCharacterisMcsandHotspots
• Results• RelatedWork
• Conclusion
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
FeatureTracking:Overview
• ProcessoflocaMngmovingobject(s)acrossframes
ApplicaMons:• CruiseControl• PedestrianTracking• InteracMveRobots
MoMonofthetrackedfeatures
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
FeatureTracking:Algorithm
Noisefilteredimage Horizontaledgeimage VerMcaledgeimage
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Highlyparallel:Eachoutputpixelcanbecomputedinparallelwithothers
ImagePreprocessing
GradientEdge
DetecMon
FeatureTracking:Algorithm
GradientEdge
DetecMon
FeatureExtracMon
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Σ
Σ
Σ
Sort
Features:{x,y,strength}EdgeImages AreaSum
ImagePreprocessing
FeatureTracking:Algorithm
GradientEdge
DetecMon
FeatureExtracMon
FeatureMatching
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Frame1Features
Correlated,withVector
ImagePreprocessing
<x,y,strgth>
<0,0,25><…>
<x,y,strgth>
<6,6,21><1,1,12><0,2,22>
SelectNearbyFeatures(e.g.,withinarangeof5x5)thenpickonewithcloseststrength.
Frame2Features
FeatureTracking:Algorithm
1 2 4Pe
rce
nta
ge
Exe
cu
tio
n T
ime
0
10
20
30
40
50
Feature Tracking
Gradient
GaussianFilter
IntegralImage
Areasum
RelaMveinputsize1X,2X,4X‐(12k,25k,100kpixels)
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
GradientEdge
DetecMon
FeatureExtracMon
FeatureMatching
ImagePreprocessing
(ImagePreprocessing)
(FeatureExtracMon)
(EdgeDetecMon)
DisparityMap:Overview
• ComputesrelaMveposiMonsofobjectsinascenecapturedbystereocameras–depth
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
DisparityMap:Algorithm
1 2 4Pe
rce
nta
ge
Exe
cu
tio
n T
ime
0
10
20
30
40
50
60
70
Disparity Map
Correlation
IntegralImage
Sort
SSD
RelaMveinputsize1X,2X,4X
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
ImagePre‐Processing
Correlatelesandrightpairs
FindDisparityMap
ImageSMtch:Overview• CombiningmulMpleimageswithoverlappingview
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
ImageSMtch:Overview
• CombiningmulMpleimageswithoverlappingview
ApplicaMons
• Moviemaking• Googleearth,GPSapplicaMonsetc
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
ImageSMtch:Analysis
1 2 4Pe
rce
nta
ge
Exe
cu
tio
n T
ime
0
20
40
60
80
100
Image Stitching
LSSolver
SVD
Convolution
NonKernelWork
RelaMveinputsize1X,2X,4X
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Imagepre‐processing
FeatureDetecMon MatchFeatures SMtchImages
(MatchFeatures)(FeatureDetecMon)(Imagepro‐processing)
ScaleInvariantFeatureTransform(SIFT):Overview
DetecMonanddescripMonofrobustlocalimagefeatures
ApplicaMons• ObjectrecogniMon• MoMonAnalysis• GestureRecogniMon
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
SIFT:Algorithm
ImageProcessing
andPyramid
GeneraMon
FeaturePyramid
KeypointDetecMon
Removelow
contrastkeypoints
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
DifferenceofGaussiansPyramid–EdgeImages
CornerdetecMon Removelowcontrastkeypoints
ImageSegmentaMon:Overview
ProcessofparMMoninganimageintomeaningfulsegments,typicallyusedtolocateobjectsandboundaries
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
ImageSegmentaMon:Algorithm
PyramidGeneraMon
and
GradientImage
ExtractconMnuousedges
Extracthighcontrastedges
Markdiscretesegments
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
UsingtheconMnuousedgeimageandhighcontrastmatrix,wemarksegments
SegmentedimageConMnuousedgeimage
SupportVectorMachine(SVM):Overview
SeparaMonofagivensetof
dataintotwocategorieswithmaximalgeometricmargin.
ApplicaMons• MachineLearning
• Neuralnetworks• DataclassificaMon
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
SVM:Algorithm
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Traintheclassifierbasedonthetraining
data
EvaluatetheclassifierbasedonthetesMng
data
QualifyofdataclassificaMonImproveswithnumberofIteraMonsusedinlearningstage.
TheexecuMonMmedependsonthenumberofdatapointsandnumberofiteraMons.
RefinetheclassifierbasedonthetesMng
data
RobotLocalizaMon:Overview
ProcessofevaluaMngpathbasedonobstaclesandasetofgoals
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
xxx
xxxx
xxxxxxx
xxxx
xxxxApplicaMons• SpaceexploraMonprograms• Autonomousvehicles/systems• MobileroboMcs
xxx
RobotLocalizaMon:Algorithm
GlobalPosiMonEsMmaMon LocalPosiMonTracking
ProbabilisMcmodelstoesMmateposiMon
1 2 4Pe
rce
nta
ge
Exe
cu
tio
n T
ime
0
10
20
30
40
50
60
70
Robot Localization
ParticleFilter
Sampling
NonKernelWork
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
(Tracking)
(EsMmaMon)
TextureSynthesis:Overview
ConstrucMnglargeimagesfromsmallimageusingstructuralcontent
ApplicaMons• Moviemaking• Graphics• ComputaMonalPhotography
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
TextureSynthesis:Algorithm
ImagePre‐processing
TextureAnalysis TextureSynthesis
text Reg. nearReg. Irreg. nearStoch.Pe
rce
nta
ge
Exe
cu
tio
n T
ime
0
10
20
30
40
50
Texture Synthesis
Sampling
ConjugateMatrix
PCA
NonKernelWork
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
(TextureAnalysis)(TextureAnalysis)(TextureSynthesis)
FaceDetecMon:Overview
DetermineslocaMonsandsizesofhumanfaces
ApplicaMons• Objecttracking• DigitalCamera,Facebook• Contentbasedimageretrieval
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
Results:ExecuMonTimes
Benchmark 12kpixels 25kpixels 100kpixels
Featuretracking 2.77 5.0 19.4
DisparityMap 0.8 1.8 6.2
ImageSMtch 0.7 10.1 23.4
SIFT 17.4 44.5 131
TextureSynthesis 18.5
ImageSegmentaMon 8.3 9.2 8.4
SVM UptoThousands
SecondsperFrame
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
TheseMmesarefor1or2framesofcomputaMononasinglecore.
LotsofpotenMalformakinguseofmulM‐coreprocessors!Farenoughawaythatwecanuselotsofcores,butcloseenoughthatit’saIainablewithmorecores.
Results:ParallelisminVisionKernelsParallelismcalculaMon:‐ WeimplementedatransformaMonpassin
LLVMinfrastructuretrackoperandsthroughtheprogramanddeterminethelongestdependencechainthroughmemory,controlandinstrucMondependences.
‐ Parallelism=#Instrs/CriMcalPathLength
Benchmark Kernel Approx.Parallelism
Disparity CorrelaMon 502
IntegralImage 160
Sort 1,700
SSD 1,800
FeatureTracking Gradient 71
GaussianFilter 637
IntegralImage 1,050
AreaSum 425
MatrixInversion 171,000
SIFT SIFT 180
InterpolaMon 502
IntegralImage 16,000
ImageSMtch LSSolver 20,900
SVD 12,300
ConvoluMon 4,500
SVM MatrixOps 1,000
Learning 851
ConjugateMatrix 502
Thebenchmarksshownontherighthavelotsofparallelism!
(Ifyouarecurious,weencourageyoutogoaheadanddownloadourbenchmarksuite!)
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
RelatedWork:Othervisioncodes
• Performance‐orientedbenchmarks– PARSEC:BodyTracking– MediaBench–image/videocompressionalgorithms– Spec2000facerec
• Accuracy‐orientedbenchmarks(forvisionresearch)– BerkeleySegmentaMonDatabaseandBenchmark
– PEIPA– Muscle
– ImageCLEF
• VisionLibraries– OpenCV‐Highlytunedvisiontoolbox
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
RelatedWork:IntelOpenCVvsSD‐VBS
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
OpenCV SD‐VBS
Goal Tuned,Feature‐loadedimplementaMonsforcommercialandacademicvisionapplicaMons
“Pure”versionsforeasyanalysis,transformaMonandparallelizaMoninmulM‐corearchitectureresearch
SourceCode C++ CorMATLAB,yourchoice
Pla\ormspecificop.miza.ons
HighlyTunedforbestperformanceonIntelandsupporMngarchitectures
AcMvelyremovedopMmizaMonsthatincreasecodecomplexity
Codingstyle
Example:FILTER
Highlyflexible;fullofopMonstochangebehaviorofeachfuncMon
Cleancodewithoutfeaturesthatdeteranalysis
2000linesofcode 50linesofcode
204condiMonalstatements NocondiMonalstatements
PointerOperaMons ArrayOperaMons
Easeofanalysis Harder Easier,becausesimplerimplementaMons
Conclusions
• ComputerVisionisanexciMngdomainwithimmensepotenMal
• Visionalgorithmsarefullofparallelism,andcanbenefitfromprocessorswithgreaterandgreaterperformance;whichmakethemidealformulM‐core
• SD‐VBSisacomprehensiveandcleanbenchmarksuiteforvision,wellsuitedformulM‐coreandmany‐coreresearch.
Publicreleasehttp://parallel.ucsd.edu/vision
Intro B1 B2 B3 B4 B5 B6 B7 B8 B9 Results RelatedWork Conclusion
WewouldliketothankthefollowingVisionresearchers:
• AndreaVedaldi• AthanasoisNoulas• TomMitchell• JavierPorMlla• EeroSimoncelli• JianboShi
ForcontribuMngtheircodeanddatasetstothebenchmarksuite.
SupportedbyNSFCAREERAward0846152
http://parallel.ucsd.edu/vision