a framework for multi-resolution and multi-touch systems

A Framework for Multi-resolution and Multi-touch Systems

Presenter: 胡婷婷多重解析度及多重觸控互動系統之開發

Advisor: 洪一平教授

A Framework for Multi-Resolution & Multi-Touch Systems of 41

Outline

Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work


Introduction

• Motivation Multi-resolution• Variable-resolution nature of human vision• Cost-effective large high-res display

Multi-touch• Intuitive interaction• Computer-vision based approach

Software Toolkit• Hide engineering challenges


Introduction

• i-m-Top interactive multi-resolution tableTop• interactive: multi-touch• multi-resolution: fovea + peripheral projectors

• i-m-Top SDK Toolkit for rapid prototyping applications• Featured on multi-resolution/multi-touch


• Overview

Introduction

ProjectorsTouch Detection

Other sensors

Cams

User Interface

i-m-TopSDK

Image Processing

Applications

Pan-tilt Unit

Multimedia


Outline



Related Work

• Multi-Resolution Display Fovea Plus Context • LCD screen (fixed) + projector

Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01


Related Work

• Multi-Resolution Display Fovea-Tablett• Tablets as high-res displays

• Tracked with visual marker• Multiple high-res is allowed

• Cons: physical boundary

J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea-�tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007


Related Work

• Software Development Toolkit DiamondTouch SDK (C++)• Multi-touch, multi-user

Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002


Related Work

• Software Development Toolkit DiamondSpin (Java): • Around-the-table interaction

• Real-time polar to Cartesian transformation

public + personal work areas rotatable circular tabletop

Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004


Related Work

• Software Development Toolkit T3 (Java)• Multiple projectors → single high-res display• Mixed-presence collaboration

Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.


Outline

Introduction Related Work i-m-Top System

• Hardware Configuration• Touch Detection

i-m-Top SDK Application Conclusion & Future Work


i-m-Top System

Peripheral Projector

Fovea Projector

IR Camera

IR Illuminator

Mirror

Pan-tilt Unit

56”96 ppi

24 ppi


i-m-Top System


• Touch Detection

i-m-Top System

Background Subtraction

OpeningDown-scale

Opening

ConnectedComponent

Palm Tracking

ConnectedComponent

PCA Analysis

Palm-fingerAssociation

Finger TrackingConnected

ComponentPCA

AnalysisPalm

Association

FingerFusionBackground

Subtraction

-

-Opening

-

finger palm


Outline

Introduction Related Work i-m-Top System i-m-Top SDK

• Design Principles• Architecture• Implementation

Application Conclusion & Future Work


i-m-Top SDK

• Design Principles Efficient prototyping

• Hide engineering details• Use metaphors• Provide reusable components

Touch-detection module independent• Vision-based & electronic-based approaches

Real-time requirement• Touch event handling + UI rendering

Extensibility• Integrate with other components


i-m-Top SDK

• ArchitectureVirtual Director

Scene

ViewNode

Scene

ViewNode

Scene Scene

Node Node Viewport /Virtual Camera

Position / Textures /

Event Handlers /Animation Effects

Global Coordinate

Local Coordinate

Node

Node Virtual Camera (Fovea View)

Virtual Camera (Peripheral View)


i-m-Top SDK

• Architecture

SceneTouch

Detection

View

Application

Node

OtherSensors

Projector

Projector

PTU

Speaker

Socket

Window Message

COM Message Parser OpenGL c#

DeviceControl

Virtual Director

Event queue

LED LightCamNode


i-m-Top SDK

• Loop of virtual director

Check Inputs

Update States

Render Scenes

Multimedia Output

Affiliated Timer Functions

Sleep1/fps

Time spent

1/fps-T

T

0


i-m-Top SDK

• Sample Code A scene with a photo

InitScenes( ){ oScene scene = new oScene(); //construct a scene director.AddScene(scene); //add scene to director

oView view = new View(0, 0, 1280, 720); //construct a viewscene.AddView(view); //add view to scene

oPhoto photo = new oPhoto(0,0,-100,640,480,”sun.png”); //construct a photoscene.AddNode(photo); //add to scene

}


i-m-Top SDK

• Implementation Multi-resolution Handling Touch-event Processing Reusable Components


Implementation

• Multi-resolution Handling Calibration• Homography of fovea & peripheral projections• Run-time warping

Masking Steerable fovea projection• Region/Object oriented

Multi-presented Information


• Calibration Homography of fovea & peripheral projections• Provide tools for manual calibration• Save homography/ center point /pan-tilt of PTU

Implementation

F1 F2

F3F4

V1 V2

V3 V4

Y

X

V2V1

V3 V4

P1

P4

P2

P3

Y

X

F1 F2

F3F4

V1 V2

V3 V4


• Calibration Run-time warping• Adapt OpenGL rendering pipeline

Implementation

Projection Matrix

Perspective Division

Viewport Transformation

Model-View Matrix

Local coordinate

Global coordinate

Eyecoordinate

Clipping coordinate

Normalized device

coordinate

Windowcoordinate

Warping Homography

Matrix


• Masking

Implementation

foveaperipheral fovea


• Masking Find maximum region in fovea• Utilize Stencil buffer in OpenGL

Implementation

Applyarbitrary masks

Set ‘0’ in mask regions

Set ‘1’ in the fovea region

Invert pixels in fovea region


Implementation

• Steerable Fovea Projection Region-oriented

• Spotlight on a specific region • Steered by finger gestures

Object-oriented• Spotlight on a specific object• Move with object

Real-time update • Steer the mirror• Retrieve homography• Apply mask


• Multi-presented Information Different content in fovea and peripheral regions• Eg. Google Map• Automatically change

Implementation

oRectangle multi_presentation = new oRectangle(0, 0,-100, 200, 200); multi_presentation.LoadTexture(“context.png"); //load low-resolution image multi_presentation.LoadHighTexture(“detail.png"); //load high-resolution image


• Touch-event Processing

VirtualDirector

Implementation

TouchDetection

Pass

MessageParser Node

i-m-Top SDKTouch events

Receive Check Dispatch Handle (by event handler)


Implementation

• Touch-event Processing Receive & parse• Run on background threads

Pass to director• Gather all events in one detection frame before pass

• if rendering rate < detection rate» Keep vital events: ‘Up’, ‘Down’» Discard other events: ‘Move’

VirtualDirector

Pass

MessageParser Node

i-m-Top SDK

Receive


Implementation

• Touch-event Processing Check• ‘Down’: Test if hit some node

• By SELECTION mode in OpenGL• If yes: add to TargetTable (event id → node)

• ‘Move’, ‘Up’: Check if there is a target• Hit TargetTable by event id• ‘Up’ : Remove from TargetTable

Dispatch• Dispatch events to each node

VirtualDirector

Dispatch

MessageParser Node

i-m-Top SDK

Check


Implementation

• Touch-event Processing Define event handler• Node decides how to act• Default handler

• Move/Scale/Rotate under ‘Move’ events• Redefine handler

InitScenes( ){ //……

//add new event handlers to the photo object photo.MultiFingerEvent += new MultiFingerHandler(MyMultiFingerHandler); photo.FingerClick += new FingerClickHandler(MyFingerClickHandler);}

VirtualDirector

Handle

MessageParser Node

i-m-Top SDK


P4

Rotation: R

Implementation

P1

P2

P3

P4

Translation: T Scaling: S

P12

P3

P4

P3

P34

P3

P4

P1

P2


Implementation

• Reusable components Multimedia• Video/audio• Camera capture

UI components• Virtual keyboard• Text-pad• Button, scrollbar…


Outline



Application

• Map Browser Different content in the fovea & peripheral

• Photo Browser Object/Region oriented fovea projection

• Emotion Player Interactive art

c


Outline



Conclusion

• i-m-Top Multi-resolution

• Accommodating to human vision• Cost-effective large high-res display

Multi-touch• Intuitive interaction

• i-m-Top SDK Hide engineering challenges

• Rapid prototyping applications• Investigating new interfaces / interactions


Future Work

Attentive computing• Combine with gaze-tracking

3D interaction• Surface to space

Auto calibration• Vision-based• Continuous pan-tilt / homography map

Thanks for your listening

a framework for multi-resolution and multi-touch systems

Documents