binocular stereo vision
DESCRIPTION
Binocular Stereo Vision. Region-based stereo matching algorithms Properties of human stereo processing. Solving the stereo correspondence problem. Measuring goodness of match between patches. (1) sum of absolute differences. Σ | p left – p right |. Optional: divide by - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/1.jpg)
CS332 Visual ProcessingDepartment of Computer ScienceWellesley College
Binocular Stereo Vision
Region-based stereo matching algorithms
Properties of human stereo processing
![Page 2: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/2.jpg)
Solving the stereo correspondence problem
1-2
![Page 3: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/3.jpg)
1-3
(1) sum of absolute differences
Σ | pleft
– pright
|
(2) normalized correlation
(pleft
– pleft
) (pright
– pright
)
patch
σpleft σ
pright
Σ patch
Measuring goodness of match between patches
(1/n)
(1/n)
Optional: divide byn = number of pixels
in patch
![Page 4: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/4.jpg)
Region-based stereo matching algorithm
for each row r
for each column c
let pleft be a square patch centered on (r,c) in the left image
initialize best match score mbest to ∞initialize best disparity dbest
for each disparity d from –drange to +drange
let pright be a square patch centered on (r,c+d) in the right image
compute the match score m between pleft and pright
(sum of absolute differences)
if (m < mbest), assign mbest = m and dbest = d
record dbest in the disparity map at (r,c)
1-4
(normalized correlation)
How are the assumptions used??
![Page 5: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/5.jpg)
1-5
left right
The real world works against us sometimes…
![Page 6: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/6.jpg)
Example: Region-based stereo matching, using filtered images and sum-of-absolute differences
1-6
(from Carolyn Kim, 2013)
![Page 7: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/7.jpg)
1-7
Properties of human stereo processing
Use features for stereo matching whose position and disparity can be measured very precisely
Stereoacuity is only a few seconds of visual angle
difference in depth 0.01 cm at a viewing distance of 30 cm
![Page 8: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/8.jpg)
1-8
Properties of human stereo processing
Matching features must appear similar in the left and right images
For example, we can’t fuse a left stereo image with a negative of the right image…
![Page 9: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/9.jpg)
1-9
Properties of human stereo processing
Only “fuse” objects within a limited range of depth around the fixation distanceVergence eye movements are needed to fuse objects over larger range of depths
![Page 10: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/10.jpg)
1-10
Properties of human stereo vision
We can only tolerate small amounts of vertical disparity at a single eye position
Vertical eye movements are needed to handle large vertical disparities
![Page 11: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/11.jpg)
1-11
Properties of human stereo processing
In the early stages of visual processing, the image is analyzed at multiple spatial scales…
Stereo information at multiple scales can be processed independently
![Page 12: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/12.jpg)
1-12
Neural mechanisms for stereo processing
G. Poggio & colleagues:
complex cells in area V1 of primate visual cortex are selective for stereo disparity
neurons that are selective for a larger disparity range have larger receptive fields
zero disparity: at fixation distance near: in front of point of fixation far: behind point of fixation
![Page 13: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/13.jpg)
1-13
In summary, some key points…
• Image features used for matching:simple, precise locations, multiple scales, similar between left/right images
• At single fixation position, match features over a limited range of horizontal & vertical disparity
• Eye movements used to match features over larger range of disparity
• Neural mechanisms selective for particular ranges of stereo disparity
![Page 14: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/14.jpg)
1-14
Matching features for the MPG stereo algorithm
zero-crossings of convolutions with 2G operators of different size
L
M
S
rough disparities over large range
accurate disparities over small range
![Page 15: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/15.jpg)
1-15
large w
left
large w
right
small w
left
small w
right
correct match outside search range at small scale
![Page 16: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/16.jpg)
1-16
large w
left
right
small w
left
right
correct match now inside search range at small scale
vergence eye movements!
![Page 17: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/17.jpg)
1-17
Stereo images (Tsukuba, CMU)
![Page 18: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/18.jpg)
1-18
Zero-crossings for stereo matching
- +
… …
![Page 19: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/19.jpg)
1-19
Simplified MPG algorithm, Part 1
To determine initial correspondence:(1) Find zero-crossings using a 2G operator with central
positive width w(2) For each horizontal slice:
(2.1) Find the nearest neighbors in the right image for each zero-crossing fragment in the left image(2.2) Fine the nearest neighbors in the left image for each zero-crossing fragment in the right image(2.3) For each pair of zero-crossing fragments that are closest neighbors of one another, let the right fragment be separated by δinitial from the left. Determine whether δinitial is within the matching tolerance, m. If so, consider the zero-crossing fragments matched with disparity δinitial
m = w/2
![Page 20: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/20.jpg)
1-20
Simplified MPG algorithm, Part 2To determine final correspondence:(1) Find zero-crossings using a 2G operator with reduced width
w/2(2) For each horizontal slice:
(2.1) For each zero-crossing in the left image:(2.1.1) Determine the nearest zero-crossing fragment
in the left image that matched when the 2G operator width was w
(2.1.2) Offset the zero-crossing fragment by a distance δinitial, the disparity of the nearest matching zero-crossing fragment found at the lower resolution with operator width w(2.2) Find the nearest neighbors in the right image for each zero-crossing fragment in the left image(2.3) Fine the nearest neighbors in the left image for each zero-crossing fragment in the right image(2.4) For each pair of zero-crossing fragments that are closest neighbors of one another, let the right fragment be separated by δnew from the left. Determine whether δnew is within the reduced matching tolerance, m/2. If so, consider the zero-crossing fragments matched with disparity δfinal = δnew + δinitial
![Page 21: Binocular Stereo Vision](https://reader036.vdocuments.pub/reader036/viewer/2022081506/568134f7550346895d9c425b/html5/thumbnails/21.jpg)
1-21
Coarse-scale zero-crossings:
Use coarse-scale disparities to guide fine-scale matching:
Ignore coarse-scale disparities:
w = 8m = 4
w = 4m = 2
w = 4m = 2