15-463: computational...
TRANSCRIPT
Multiple View Geometry
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2007
© Martin Quinn
…with a lot of slides stolen from Steve Seitz and Jianbo Shi
Our Goal
The Plenoptic Function
P(θ,φ,λ,t,VX ,VY ,VZ )
How can we compress this into something manageable?
Stereo ReconstructionThe Stereo Problem
• Shape from two (or more) images• Biological motivation
knownknowncameracamera
viewpointsviewpoints
Why do we have two eyes?
Cyclope vs. Odysseus
1. Two is better than one
2. Depth from Convergence
Human performance: up to 6-8 feet
3. Depth from binocular disparity
Sign and magnitude of disparity
P: converging point
C: object nearer projects to the outside of the P, disparity = +
F: object farther projects to the inside of the P, disparity = -
Stereo
scene pointscene point
optical centeroptical center
image planeimage plane
Stereo
Basic Principle: Triangulation• Gives reconstruction as intersection of two rays• Requires
– calibration– point correspondence
Stereo correspondenceDetermine Pixel Correspondence
• Pairs of points that correspond to same scene point
Epipolar Constraint• Reduces correspondence problem to 1D search along conjugate
epipolar lines
epipolar plane epipolar lineepipolar lineepipolar lineepipolar line
Stereo image rectification
Stereo image rectification
Image Reprojection• reproject image planes onto common
plane parallel to line between optical centers• a homography (3x3 transform)
applied to both input images• pixel motion is horizontal after this transformation• C. Loop and Z. Zhang. Computing Rectifying Homographies for
Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
Stereo Rectification
Your basic stereo algorithm
For each epipolar lineFor each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows• This should look familar...• Can use Lukas-Kanade or discrete search (latter more common)
Window size
• Smaller window+–
• Larger window+–
W = 3 W = 20
Effect of window size
Stereo results
Ground truthScene
• Data from University of Tsukuba• Similar results on other images without ground truth
Results with window search
Window-based matching(best window size)
Ground truth
Better methods exist...
State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
Depth from disparity
f
x x’
baseline
z
C C’
X
f
input image (1 of 2)[Szeliski & Kang ‘95]
depth map 3D rendering
• Camera calibration errors• Poor image resolution• Occlusions• Violations of brightness constancy (specular reflections)• Large motions• Low-contrast image regions
Stereo reconstruction pipelineSteps
• Calibrate cameras• Rectify images• Compute disparity• Estimate depth
What will cause errors?
Stereo matchingNeed texture for matching
Julesz-style Random Dot Stereogram
Active stereo with structured light
Project “structured” light patterns onto the object• simplifies the correspondence problem
camera 2
camera 1
projector
camera 1
projector
Li Zhang’s one-shot stereo
Active stereo with structured light
Laser scanning
Optical triangulation• Project a single stripe of laser light• Scan it across the surface of the object• This is a very precise version of structured light scanning
Direction of travel
Object
CCD
CCD image plane
LaserCylindrical lens
Laser sheet
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Portable 3D laser scanner (this one by Minolta)
Real-time stereo
Used for robot navigation (and other tasks)• Several software-based real-time stereo techniques have
been developed (most based on simple discrete search)
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
Volumetric Stereo
Scene VolumeScene VolumeVV
Input ImagesInput Images(Calibrated)(Calibrated)
Goal: Goal: Determine transparency, radiance of points in VDetermine transparency, radiance of points in V
Discrete Formulation: Voxel Coloring
Discretized Discretized Scene VolumeScene Volume
Input ImagesInput Images(Calibrated)(Calibrated)
Goal: Assign RGBA values to voxels in Vphoto-consistent with images
Complexity and Computability
Discretized Discretized Scene VolumeScene Volume
N voxelsN voxelsC colorsC colors
33
All Scenes (CN3)Photo-Consistent
Scenes
TrueScene
Issues
Theoretical Questions• Identify class of all photo-consistent scenes
Practical Questions• How do we compute photo-consistent models?
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
Reconstruction from Silhouettes
Binary ImagesBinary Images
Approach: • Backproject each silhouette• Intersect backprojected volumes
Volume Intersection
Reconstruction Contains the True Scene• But is generally not the same • In the limit get visual hull
> Complement of all lines that don’t intersect S
Voxel Algorithm for Volume Intersection
Color voxel black if on silhouette in every image• O(MN3), for M images, N3 voxels• Don’t have to search 2N3 possible scenes!
Properties of Volume Intersection
Pros• Easy to implement, fast• Accelerated via octrees [Szeliski 1993]
Cons• No concavities• Reconstruction is not photo-consistent• Requires identification of silhouettes
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate3.3. Color if consistentColor if consistent
(standard deviation of pixel colors below threshold)
Voxel Coloring Approach
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
The Global Visibility Problem
Inverse Visibilityknown images
Unknown SceneUnknown Scene
Which points are visible in which images?
Known SceneKnown Scene
Forward Visibilityknown scene
LayersLayers
Depth Ordering: visit occluders first!
SceneSceneTraversalTraversal
Condition: Condition: depth order is depth order is viewview--independentindependent
Calibrated Image Acquisition
Calibrated Turntable360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
Voxel Coloring Results (Video)
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Limitations of Depth Ordering
A view-independent depth order may not exist
p q
Need more powerful general-case algorithms• Unconstrained camera positions• Unconstrained scene geometry/topology
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
Space Carving Algorithm
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Repeat until convergence
• Choose a voxel on the current surface
• Carve if not photo-consistent• Project to visible input images
Convergence
Consistency Property• The resulting shape is photo-consistent
> all inconsistent points are removedConvergence Property
• Carving converges to a non-empty shape> a point on the true scene is never removed
p
Structure from Motion
Reconstruct • Scene geometry• Camera motion
UnknownUnknowncameracamera
viewpointsviewpoints
Three approachesThree approaches
Outline of a simple algorithm (1)Outline of a simple algorithm (1)
• Based on constraints
• Input to the algorithm (1): two images
Outline of a simple algorithm (2)Outline of a simple algorithm (2)
• Input to the algorithm (2): User select edges and corners
Outline of a simple algorithm (3)Outline of a simple algorithm (3)
• Camera Position and Orientation Determine the position and orientation of camera
Outline of a simple algorithm (4)Outline of a simple algorithm (4)
• Computing projection matrix and Reconstruction
Outline of a simple algorithm (5)Outline of a simple algorithm (5)
• Compute 3D textured triangles
Facade Facade
SFMOMA (San Francisco Museum of Modern Art) by Yizhou Yu,
Façade (Debevec et al) inputs
Façade (Debevec et al)