Finger Detection for Multi-Touch Tabletop Display System

Finger Detection for Multi-Touch Tabletop Display Systems

Su-ting, Chuang2010/8/21OutlineIntroductionRelated WorksSystem and MethodExperimentConclusion & Future Work22OutlineIntroductionRelated WorksSystem and MethodExperimentsConclusion & Future Work33Introductioni-m-Topinteractive multi-resolution tableTopinteractive: multi-touchmulti-resolution: fovea + peripheral projectors

4

Chia, Yi-Wei,i-m-Top, an Interactive Multi-resolutionTabletopDisplay System (2007)labimtop system ()interactive: multi-touchmulti-resolution: fovea + peripheral projectorsFinger detectionPalm detection

4Introductioni-m-Top SDKA software development toolkit for rapid prototyping multi-resolution and multi-touch applications

5IntroductionLow computation efficiencyNon-uniform lighting problemVarious finger touch responses among different positionsNo tools available for helping users determine parameters automatically

66OutlineIntroductionRelated WorksSystem and MethodExperimentsConclusion & Future Work77Related WorksDI (Diffused Illumination)J. Rekimoto and N. Matsushita, Perceptual surfaces: Towards a human and object sensitive interactive display," Workshop on Perceptural User Interfaces (PUI'97), 1997.

8An IR camera with IR illuminators to observe hands

8Related WorksFTIR (Frustrated Total Internal Reflection)

J. Y. Han, Low-cost multi-touch sensing through frustrated total internal reflection," in Proceedings of the 18th annual ACM symposium on User interface software and technology (UIST '05). New York, NY, USA: ACM Press, 2005, pp. 115-118.99Related WorkTouchLibA multi-touch development kit

Finger detection processing flow chart10

BackgroundSubtractionSimpleHighpass

IntensityScalingThresholdingFinger AnalysisParameters to be adjusted manually ImageEnhancementa1a2, a3a4a5a6* Background subtraction: subtract value Simple highpass: highpass & median kernel size Intensity scaling: scale value Thresholding: threshold Finger analysis: finger size

background (edge)Scale thresholdthreshold10Related WorkDirectShowFilter-based framework GShowGPU-accelerated frameworkCombination of DirectX and DirectShow

11

Dshow: [media-streaming] Provide high-quality capture and playback of multimedia streams can perform high-quality video and audio playback or captureIt supports capture from digital and analog devices based on the Windows Driver Model (WDM) or Video for Windows. It automatically detects and uses video and audio acceleration hardware when available, but also supports systems without acceleration hardware.

Gshow: combination of directX and directShow

11OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work1212Hardware Configuration

(2) IR Camera(3) IR Illuminator(1) Peripheral Projector1313Hardware ConfigurationOrder of diffuser layer and touch-glass layer14

Diffuser layerIR illuminatorIR cameraspotIR illuminatorIR cameraTouch-glass layerIR cameraspotIR camera

2143

IR diffuser The surface of this kind of tabletop system is usually composed of a diffuser layer and a touch-glass layer.14Hardware ConfigurationProblem:IR rays reflected by the touch-glass will result in hot spot regions in camera viewsSolution:Two cameras to cover spot regions for each others sheltered IR spot zone

15

camera

15Software ArchitectureDetection systemImage StitchingFinger DetectionFinger Tracking

Parameter determination16ImageStichingFingerDetectionFinger TrackingImage StitchingCombine multi-camera view into a virtual camera viewFinger DetectionRecognize touched fingertipFinger TrackingSmooth the trajectory of finger and fix lost results16Software Architecture17ImageStichingFingerDetectionFinger Tracking

Enhance in GPU17Image StitchingCombine multi-camera view into a virtual camera view

18

Image StitchingRemove IR spot effectUnify finger size among different positions of tableEasily fit in existing finger detection systems19

Reduce matching problemBe compatible with existent finger detection system

19Image Stitching20ImageBlendingIR Camera(L)IR camera(R)UndistortionUndistortionHomoWarpHomoWarp

Image StitchingHomoWarp21

521625431436

2521Image StitchingImage Blending

22

22Finger DetectionTouchLib

Our method23NormalizationDifference of GaussiansBackground SubtractionThresholdingFinger AnalysisSimple HighpassScaleBackground SubtractionThresholdingFinger Analysis

Finger DetectionInsert normalization

24Finger DetectionNormalizationMethodModel distribution of IR illuminationUse specific material to simulate foregroundConstruct normalization map

Normalize foreground image

ResultBefore normalization: mean = 75, standard variation = 30After normalization: mean = 254.8, standard variation = 1.2525

Construct normalization mapCalculate each pixels dynamic range Stretch dynamic range to 0-255Normalize foreground imageMultiply normalization ma

25Finger DetectionApproximate Difference of Gaussians (DoG)Modified from simple highpass in TouchLibTouchLib: (Original image Blurred image ) + median filter

26

Simple highpass -blur, noise, touchLibmedian filter , midian filter time complexity

sigma = (n/2 - 1)*0.3 + 0.8 , n=kernel size26Fingertip TrackingGoalSmooth the trajectory of finger MethodKalman filterUse Kalman filter topredict the current position of tn

27

Predict the new state and its uncertaintyCorrect the tracker with its new measurementAssume white noise and uniform velocity

27Parameter DeterminationRequirements of ideal finger detection systemHigh sensitivity missNoise-free false alarm GoalFind an appropriate set of parameters for finger detection system fulfilling the requirements28Parameter Determination29Parameters DeterminatorParameter CombinationDetection ResultApplicable set of ParametersTestSetTouch DataGround Truth(Trace)Detection SystemParameters Determinator : generate parameter combinations & evaluate parameter combinations by calculating miss and false alarm

29Parameter DeterminationEvaluation of parametersData CollectionDepict traceMeasurementMinimize # of miss and false alarm30

requirmentsmiss & false alarm30Parameter DeterminationIdeal finger detection Only one fingertip landing on traceContinuity among frames31

Good frameOnly one finger landing on traceContinuity

31OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work3232ExperimentsPerformance evaluation33

640 x 48033ExperimentsParameter determinationDecide parameters in our systemAdopt sampling-based parameter search technique

34NormalizationDifference of GaussianBackground SubtractionBinaryFinger Analysisb4b1: subtraction valueb2: kernel size b3: thresholdb4: finger sizeb3b2b134ExperimentsParameter determinationExhaustive searchParameter combination5 (Iteration) *5 (Iteration) *5 (Iteration) *5 (Iteration) = 625Applicable parameter num16/625 = 2.56%

35Subtract valueSmooth kernelThresholdFinger sizeLow bound051010Step55510High bound2025305035OutlineIntroductionRelated WorkSystem and MethodExperimentsConclusion & Future Work3636Conclusion & Future WorkMulti-touch detection systemGPU-acceleratedNon-uniform lighting problem solvedAutomatically parameter determination tool proposed

Future WorkOptimize parameter determination

37Use more intelligent way to collect data

37Thanks for your attention38