Progress Report

An Automatic Calibration Approach for Interactive Multi-resolution Tabletop Display System1/33Speaker: Bo-Shiun Chiou

2010.05.21OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationExperimental ResultConclusion

2 /33IntroductionMotivationMulti-resolutionFeaturing nature of human-visionCost-effective large high-res displayProjector mosaicHardware/software configurationSeamless multi-projector display3 /333IntroductionIssuesGeometric CalibrationDistortion effectMisalignmentPhotometric UniformityCommon color gamutOverlapping/Inter/Intra color variationAuto/Manual ApproachCost, accuracy, stability, etc.

4 /334OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationExperimental ResultConclusion

5 /33Related WorkMulti-Resolution DisplayFocus Plus ContextProjector + LCD monitorFixed high-res areaP. Baudisch and N. Good, Focus plus context screens: visual context and immersion on the desktop, in SIGGRAPH 02: ACM SIGGRAPH 2002 conference abstracts and applications. New York, NY, USA: ACM, 2002, pp.7070.6 /33

Baudisch has combined an LCD monitor and video projector to make a focus plus context screen [Bau01] a very large monitor that has high resolution in the focal region, plus a large contextual area. This system superficially seems similar to the Escritoire because it combines a large context with a high-resolution focus, but it is purely an enlarged version of the conventional interface. Conventional programs are controlled using the standard peripherals the keyboard and mouse and no projector calibration is performed.6Related WorkMulti-Resolution DisplayFovea TablettTablets as high-res areaCoded markers for trackingSupport multiple high-resPhysical BoundaryVisual discontinuityJ. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, Fovea-tablett: A new paradigm for the interaction with large screens, in HCI (8), 2007, pp. 278287.7 /33

7

Related WorkT. T. Hu, Y. W. Chia, L. W. Chan, Y. P. Hung, & J. Hsu, (2008). i-m-Top: Aninteractive multi-resolution tabletop system accommodating to multi-resolutionhuman vision.3rd IEEE International Workshop on Horizontal Interactive HumanComputer Systems, 2008. TABLETOP 2008,177-1808 /33Multi-Resolution Displayi-m-TopFovea + peripheral projectorsMovable high-res areaMulti-touchIntuitive interactionRelated WorkAutomatic projector calibrationLight sensors embeddedStructured light patternsCover planar/non-planar surfaceSimple and accurateLee, J., Dietz, P., Aminzade, D., Raskar, R., and Hudson, S. "Automatic ProjectorCalibration using Embedded Light Sensors", Proceedings of the ACM Symposiumon User Interface Software and Technology, October 2004.

9 /33

Calibrating onto non-planar surfaces is complex, has high computational requirements, and the camera-projector geometry must be precisely known and remain rigid such that even changing the projectors zoom may be problematic. By embedding optical sensors in the target surface at the points of interest and measuring the light from the projector directly, the majority of these problems disappear. The problem becomes quite simple, and the result is accurate and robust.9Related WorkAutomatic projector calibrationCamera homography treesForm single virtual cameraScalable multi-projector displayH. Chen, R. Sukthankar, G. Wallace, T.-J. Cham, Calibrating Scalable Multi-Projector Displays Using Camera Homography Trees, CVPR Technical Sketch, 2001.10 /33

This becomes a difficult computer vision problem subject to camera resolution, reflective properties of the target surface, lighting conditions, and background separation.

One cannot directly compute a homography between two camera views that do not overlap since they share no point correspondences. Therefore, our system builds a tree of homography relationships between adjacent views that spans thecomplete set of cameras1; the mapping from any given camera to the panoramic reference frame is determined by compoundingthe homographies along the path to the reference view at the root of the tree10Related WorkAutomatic projector calibrationArray of ARTag MarkersTime-savingM. Fiala. Automatic Projector Calibration Using Self- Identifying Patterns. InProceedings of the 2005 IEEE Computer Society Conference on Computer Visionand Pattern Recognition (CVPR 05) - Workshops (PROCAMS 2005), volume3,page 113. IEEE Computer Society, 2005.

11 /33Our method described hereing uses self-identifying patterns to reduce the number of image frames that need to be displayed on each display element, quite often only two or three calibration patterns per display element. This results in a faster autocalibration time as opposed to binary encoding techniques that need to display more calibration patterns. Our concentrates on planar display surfaces byfinding homographies instead of projection matrices, but the method can be extended to three-dimensional display surfaces.11Related WorkAutomatic projector calibrationContinuous Self-CalibrationRecover 3D model of surfaceJ. Zhou, L. Wang, A. Akbarzadeh, and R. Yang, Multi-projector display with continuous self-calibration. In Workshop on Projector- Camera Systems (PROCAMS), 2008.12 /33

To solve the geometric problem, in [8], projected patterns were brought in conjunction with two calibrated cameras to recover a 3-D model of an arbitrary surface,thus the relative 3-D geometric relationship (positions and orientations) between projectors and the display surface will be constructed.12OutlineIntroductionRelated WorkSystem DesignHardware ConfigurationSoftware ArchitectureProjector and Camera CalibrationExperimental ResultConclusion

13 /33Hardware Configuration

(1) Peripheral Projector(2) Fovea Projector(6) IR Camera(5) IR Illuminator(3) Mirror(4) Pan-Tilt Unit

(7) Color Camera14 /33OutlineIntroductionRelated WorkSystem DesignHardware ConfigurationSoftware ArchitectureProjector and Camera CalibrationExperimental ResultConclusion

15 /3315Software Architecture16 /33Status ControlPattern SelectRenderingProjectorUn-distortionPre-ProcessingCandidate Point SelectionCameraApplicationsApplicationEncodingDecodingMapping FunctionAnalysisOutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationPeripheral Projector CalibrationFovea Projector CalibrationIR Camera CalibrationExperimental ResultConclusion

17 /3317Peripheral Projector CalibrationPP

Camera View :

HCamPPHCamTableHPPTableHPPTable=HCamPPHCamTable-1Use gray code patterns

18 /33

Pre-warped imagesteerable projectorpatterncornerlinefit->Patternpaper!:average

18Peripheral Projector CalibrationHomography(Cam, Table)Mean ShiftEdge DetectLine FittingCorner ID

C0C1C2C3MappingPeripheral Projector CalibrationPreprocessPoint SelectEncodingDecodingMaskingContrastStretchDenoiseMapping

C0C1C2C3...

P0P1P2P3OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationPeripheral Projector CalibrationFovea Projector CalibrationIR Camera CalibrationExperimental ResultConclusion

21 /33Fovea Projector CalibrationFP

Camera View :

HCamFPHCamTableHFPTableHFPTable=HCamFPHCamTable-1

22 /33Use gray code patterns

Pre-warped image

PP22OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationPeripheral Projector CalibrationFovea Projector CalibrationHomography Estimation using Lookup TablesIR Camera CalibrationExperimental ResultConclusion

23 /33Homography Estimation using Lookup Tables24 /33Interpolate (Pan, Tilt) for new point NP0P1T0T1d0d1d2d3

YXTableHomography Estimation using Lookup Tables25 /33Interpolate candidate pointP0P1T0T1NPNTYXTableOutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationPeripheral Projector CalibrationFovea Projector CalibrationIR Camera CalibrationExperimental ResultConclusion

26 /33IR Camera CalibrationSemi-automatic approachFinger detectionEasy & user-friendlyPP

IR Camera

27 /3327OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationExperimental ResultConclusion

28 /33Experimental ResultComparison-manual approachGoogle mapIncomplete fovea regionTake about 60 mins for 36 calibration poses

29 /33F1F2F3F4V1V2V3V4YXF1F2F3F4V1V2V3V4: Calibration Pose: Touch Point

29Experimental ResultComparison-automatic approachGoogle mapComplete fovea regionTake about 40 mins for 290 calibration poses30 /33

30OutlineIntroductionRelated WorkSystem DesignProjector and Camera CalibrationExperimental ResultConclusion

31 /33Conclusion32 /33We provide an automatic calibration method in multi-resolution display with vision-based techniques

Continuous pan-tilt/homography mapThank you33 /33