luc mertens + rudi penne + wim abbeloos master ... · pdf filekarel de grote...
Post on 26-Feb-2018
230 Views
Preview:
TRANSCRIPT
Karel de Grote University College
Luc Mertens + Rudi Penne + Wim Abbeloos
Master Electromechanics, Industrial Automation
Studenten: Masters EMAU + Erasmus + Projecten …
Research: Tetra, KHLim, Phaer, DataVision, Egemin, Hero, IFM, …
September 2012 www.innovatienetwerk.be/projects/1583
Industrial
Luc.mertens@kdg.be
wim.abbeloos@kdg.be
Innovatie door
Wetenschap en Technologie
03D2xx-cameras PMD[Vision]CamCube 3.0 Fotonic RGB_Z70
van IFM-Electronics Highest Res = 352 x 288 pix! 160 x 120 pixel
(64x50 pix.)
Swiss Ranger MESA 170 x 145 pix. DepthSense 320
( 320 x 240 pix )
‘Time of Flight’ camera types .
Melexis: EVK75301 80x60 pix.
j
i
I
1 1
J
u.ku
v.kv
d
r D
x
y
z
(0,0,f)
f
R
phi
u = j – J/2 ; uk = ku*u v = i – I/2 ; vk = kv*v tg φ = uk/f r = √(xu²+f²) d = √(uk²+vk²+f²) D/d = x/uk = y/vk = z/f
J/2
I/2
ToF VISION: world to image, image to world - conversion
N
Aan elk wereldpunt en aan zijn locale omgeving kunnen 12 belangrijke coördinaten verbonden worden:
x, y, z, Nx, Ny, Nz, N, kr, kc, R, G, B .
kr ,kc
A
f
a P
vt
yPc
zPc
zw
yw
1. Camera x // World x // Robot x
2. World (yw,zw) = Robot (yr,zr) + ty
3. yPw = -(z0 – zPc).sin(a) + yPc.cos(a) = yPr + ty
4. zPw = (z0 – zPc).cos(a) + yPc.sin(a) = zPr
a
Material handling
Work plane = reference.
yr
zr
ty
Mitsubishi
Pepper handling
Connected peppers can be
distinguised by means of local
gradients. Gradients can
quickly be direved from
d/D-ratios.
'PepperD_short.m'.
Stitching two or more ToF-images.
Double the angle of view / generalized binocular view.
z0
Z
zR
zL DL DR
xR
X
z0 is the intersection distance of both symmetrical optical axes.
XL = xL.cos(a) – (zL+z0).sin(a)
XR = xR.cos(a) + (zR+z0).sin(a)
ZL = xL.sin(a) + (zL+z0).cos(a)
ZR = -xR.sin(a) + (zL+z0).cos(a)
a- a+
Straightforward calibration.
Options: - alternated image triggering (left/right)
- simultaneous triggering but with different modulation frequencies .
Cameleon View
Other experiments with ToF cameras
Real images of
a sewer at KdG.
Stitched volume
(6x7 images).
Dimensions
in meter.
Stitching ToF-images.
Outdoor ToF experiments: road quality analysis. VIM.m
www.TriDiCam.net
Line sensor
The first line sensor with a
resolution of 64 x 2 pixel
to be developed by
Fraunhofer IMS.
15 kHz scan speed.
3 mm x 10000 Hz 30 m/s
108 km/u @ 10000 Hz 3 mm.
Pixel shape
Motion Blur
- deblurring is structural incompatible with noise (see: Wiener Filters) !
- the unstability can be explained from matrix algebra (patent iVL).
- instability reduction is possible to some extend (regularisation).
- the better we can handle ‘blur’ in real time the faster objects may
flow ‘through’ the ToF-images. (VIM-project).
- Motion Blur in ToF-images needs further investigations… PhD ?
Which pixels have ‘seen’ a
specific dice eye and for what
integration time.
Motion deblur for ‘color
cameras ’ and for ‘ToF cameras’
ask for different approaches.
Speed: v = 100 mm/sec ; FoV = 100 pixel/1000 mm.
Pixel speed = v*FoV = 10 pixel/sec .
Sample rate: f = 10 Hz ~ Edge displacement = 1 pixel/sample.
z
y = v.t
Closer to the
ToF camera
Further away
Closer
Further away
from the ToF
camera
Speed evaluation of a ToF camera 1/2.
Blurred v.Ti
Linear displacement during
the integration time Ti :
Δs = v.Ti mm.
e.g. FoV = 100 pixel / 1000 mm. Linear edge displacement:
Δp = Δs . FoV pixel.
‘Moving edge’-
detection
v
Speed: ω = 1 rad/sec (± 60°/sec) ; Radius = 300 mm. Factor = 50 pixel/600 mm.
Tip speed: Vmax = ω.Rmax = 300 mm/sec. Tip pixel speed = V*Factor = 25 pixel/sec .
Sample rate: f = 10Hz ~ Tip displacement = 30 mm/sample ~ 2.5 tip pixel/sample.
z
ω.t
360° 180° 0°
Closer to the
ToF camera
Further away
Closer
Further away
from the ToF
camera
Speed evaluation of a ToF camera 2/2.
Blurred ω.Ti
Angular displacement during
the integration time Ti :
Δα = ω.Ti rad.
Tangential pixel displacement:
Δp = R.Δα.Factor pixel.
Spinning Wheel
Tetra: time to bridge
RGBd, ToF !
European experts
PMD-technologies
SoftKinetic
MESA
ifm electronic
Fotonic
Canesta
Zess Uni-Siegen
DAGM
VISICS
ETH Zurich
MUN
Metrilis
Vision++
TriDiCAM
Omec Beckon
European Projects:
Artts
Lynkeus
3D4YOU
Moses
IURO
… … … …
Who can use it ?
Who is waiting for it ?
Robotics
Navigation (AGV, Wheelchair)
Automotive
Material Handling
Automated Harvesting
Industrial Production
Bin Picking
Gesture Analysis
Human Machine Interfaces
Domotics
Gaming
Augmented reality
Security
Healthcare
Supervision on elderly people
Industry – SME
Teachers – Students
Professionals … … …
2.d = λmod . [φ/2π] ___
Error = k. λmod / (√ n . S/N )
Light takes about 66 picoseconds to travel over 1 cm.
λmod = 15 meter ; fmod = 20 MHz
λmod = 10 meter ; fmod = 30 MHz
Time of Flight
Cameras
c = λmod * fmod
λNIR = +/- µm.
Photonic Mixing Device PMD
A
Lambert : I(θ) = Imax.cos(θ) and Reflection factor BRDF (1965)
BRDF stands for: ‘bidirectional reflectance distribution function’
Each point of a surface has a normal direction n. Incident light from a direction ωin is scattered in all directions ωscatter . BRDF describes the fraction f(ωin , ωscatter) between incident and scattered light energy.
‘Lambertian’ reflection law Random texture
ωscatter
ωin
Time of Flight Cameras: error types
External error factors:
- surfaces with a low reflectivity (e.g. black and/or rough objects,..)
- unexpected NIR light sources (e.g. sun light, heated objects, ..)
- a low angle of incident ( e.g. the walls in a tunnel, .. ).
- saturation effects caused by bright reflections ( e.g. plastic, metals,.. ).
- scattering and absorption by dust, smoke or fog.
- Multiple reflections – Multi path .
- Ambiguity range (‘back-folding’).
Internal error factors:
- non-harmonic and/or inhomogeneous LED light (spatial and over time).
- gamma factor & saturation effects of CMOS (non linear integration).
- shot noise (specle), discretization noise, dark currents.
- distortions and blur caused by lenses or moving objects.
f = 80.0
f = 140.0
A simple calibration step gives the intrinsic ToF-camera parameters: ku, kv, f, ( + distorsion parameters ).
v
u
The calibration of a flat plane: e.g. a robot work plane
Resizing
+ filtering.
Temporal,
spatial
filtering.
Calibration: based on the
perception of a flat plane:
• Select a rough substrate.
• Evaluate the row and column wise
standard deviations as a function of
the integration time.
• Eliminate (if present) the systematic
odd-even-errors.
• Make use of an adequate filter:
Gaussian, Mean, Median, …
• Calculate the best fit reference
plane: z = a + b.x + c.y .
• Find the camera rotation angles:
ROTx and ROTy .
• Find the central distance z0.
Calibration = shoot some images,
calculate the parameters…
At that moment the camera is
ready for use: … ToF !
f , ku , kv
z
nx ~ f.(D4/d4 - D3/d3)/(D4/d4 + D3/d3)
ny ~ f.(D2/d2 - D1/d1)/(D2/d2 + D1/d1)
nz ~ - (u.nx + v.ny + 1 ) ▪
n d
D
d² = u² + v² + f².
O
3
1
4
2
The world normal vector at a
random position: n (nx, ny and nz) .
D/d-related calculations. Fast calculations !!
Nx,Ny ~ d(D/d)x,y / (D/d)
f
u
v
Second differences.
D-related calculations. Rapid calculations !!
Fast flatness analysis.
D1/d1 = X1/j1 ; D2/d2 = X2/j2 ; D3/d3 = X3/j3 . Take e.g.: X1= X2 = X3
- D1 + 2D2 - D3 = ( -d1/j1 + 2d2/j2 - d3/j3 ) X1 . Del2(D) ≠ 0 !
-D1 + 2D2 - D3 = [ (-1/sin1 + 1/sin2) + (1/sin2 - 1/sin3 )] X1 ≠ 0 ! Del2(d/D) =
0 !
≠
≠
f
j1 j2 j3
d1 d2 d3
D1
D2
D3
x
z
f
j1 j2 j3 j4 j5
d1 d2 d3 d4
D1
D2
D3
D4
x
z AC/AD ρ = --------
BC/BD
(j1-j4)/(j1-j5)
ρ = -------------- = 9/8
(j2-j5)/(j2-j5)
x1
Cross Ratios.
Top View Top View
A
B
C
D
Cross Ratios are
invariant under
projections !!
D1² + D3² - 2.D1.D3.cos(D1,D3) D1/D3 + D3/D1 – 2.cos(D1,D3)
D1² + D4² - 2.D1.D4.cos(D1,D4) D1/D4 + D4/D1 – 2.cos(D1,D4)
=
D2² + D3² - 2.D2.D3.cos(D2,D3) D3/D2 + D2/D3 – 2.cos(D2,D3)
D2² + D4² - 2.D2.D4.cos(D2,D4) D2/D4 + D4/D2 – 2.cos(D4,D2)
ρ² =
1 2
3 4
2.cos(D1,D3) = 2.cos(d1,d3) = [ d1² + d3² - (j3-j1)² ] / d1d3 2.cos(D1,D4) = 2.cos(d1,d4) = [ d1² + d4² - (j4-j1)² ] / d1d4
2.cos(D2,D3) = 2.cos(d2,d3) = [ d2² + d3² - (j3-j2)² ] / d2d3
2.cos(D2,D4) = 2.cos(d2,d4) = [ d2² + d4² - (j4-j2)² ] / d2d4
Cross Ratios in practice.
D1
D2
D3
D4
d4
d2 d3
d1
j1 j2
j3 j4
[D1//D3 + D3/D1 – 2.cos(d1,d3) ].[D2//D4 + D4/D2 – 2.cos(d2,d4) ]
[D1//D4 + D4/D1 – 2.cos(d1,d4) ].[D2//D3 + D3/D2 – 2.cos(d2,d3) ]
ρ² =
sensor
Optical centre
ρ² = (j3 – j1)²/(j4 – j1)²
(j3 – j2)²/(j4 – j2)² = (9/16)/(4/9) = 81/64
ρ = 9/8 For equidistant pixels:
Fast Flatness Analysis
The cosine values can be calculated in advance !
d/D-related calculations. Rapid calculations !!
Collinearity and coplanarity: ToF !
d1/D1 - d2/D2 + d3/D3 - d4/D4 = 0 (Coplanarity constraint for regular grids).
(d1/D1).[(u2v3 – u3v2) + (u3v4 – u4v3) + (u4v2 – u2v4)] -
(d2/D2).[(u1v3 – u3v1) + (u3v4 – u4v3) + (u4v1 – u1v4)] + (Coplanarity
(d3/D3).[(u1v2 – u2v1) + (u2v4 – u4v2) + (u4v1 – u1v4)] - Constraint).
(d4/D4).[(u1v2 – u2v1) + (u2v3 – u3v2) + (u3v1 – u1v3)] = 0
d1/D1 – 2.(d2/D2) + d3/D3 = 0 . (Collinearity Constraint for a regular grid:
del2(d/D) = 0 ! u1 = u2-1 ; u3 = u2+1 ).
?
d/D-related calculations. Rapid calculations !!
Coplanarity based calibration = finding f , corr, kv and ku .
d1/D1 + d3/D3 = d2/D2 + d4/D4 (Coplanarity constraint for regular grids
Dimensionless variants are possible ).
1
2
3 4
Coplanarity constraint for regular grids
d1/D1 + d3/D3 = d2/D2 + d4/D4 ( row / column balance ). dD = d./D; filter = [0 0 1 0 0 ; 0 0 0 0 0 ; -1 0 0 0 -1 ; 0 0 0 0 0 ; 0 0 1 0 0]; Flat = conv2(dD,filter,'same'); bwFlat = abs(Flat) < 1 & ~bwFloor;
Note: for a color camera when three points are 3D-located a fourth point is bounded if it is known to be in the same plane.
d/D-related calculations. Rapid calculations !!
MESA SR 4000.
Coplanarity check in a ToF image: fast and easy .
Wall
Floor
Wooden plates
Corridor 20 m.
D/d-related calculations.
For navigation purposes, the free floor area can be found from:
di/Di = e/E = [ f.sin(a0) + vi.cos(a0) ] / E .
The points near the floor fulfil the balance: E = (Di/di)[ f.sin(a0)+ vi.cos(a0)] .
Note: for a color camera ‘floor points’ fulfil: Di = E.di/[ f.sin(a0)+ vi.cos(a0)]
a0
vi
f
E di
e
Di
Camera bounded
parallel to the floor.
Floor.
camera
zi
Camera
inclination = a0 .
MESA SR4000
Case:
Bin picking
H E R O X
HERO X
Time of Flight camera
2011
YouTube: KdGiVL
Bin picking & 3D-OCR
Analyse ‘blobs’ one by one.
Find the centre of gravity XYZ, the normal direction components Nx, Ny, Nz ,
the so called ‘Tool Centre Point’ and the WPS coordinates.
YouTube: KdGiVL
Basic tasks of ToF cameras in order to support Healthcare Applications:
• Guide an autonomous wheelchair along the wall of a corridor.
• Avoid collisions between an AGV and unexpected objects.
• Give warnings about obstacles (‘mind the step’…, ‘kids’, stairs…)
• Take position in front of at a table, a desk or a TV monitor.
• Drive over a ramp at a front door or the door leading to the garden.
• Drive backwards based on a ToF camera (find the free floor).
• Park a wheelchair or an AGV autonomous.
• Command a wheelchair to reach a given position.
• Guide a patient in a semi automatic bad room.
• Support the supervision over elderly people.
• Fall detection of (elderly) people.
Healthcare Center (UA). Bart Geraets & Jan Steckel.
Imagine a wheelchair user likes to do 8 typical activities at home:
1. Sitting on the table. 2. Watching TV. 3. Looking through the window.
4. Working on a PC. 5. Reaching the corridor 6. Command to park the
wheelchair. 7. Finding some books on a shelf. 8. Reaching the kitchen
and the garden.
MESA
SR 4000
Potential driven navigation
- a known map
- ‘square waves’
- distance to end point
conform to map constraints
blue = end point
- a known map
- distance to nearest object
conform to object constraints
blue = nearby
Target point No local minima Repulsive potential
Path finder based on
ToF measurements &
guided potential flow.
Path building is possible
by means of the
‘Dijkstra Algorithm’.
ToF - RGB
correspondency
x
y
IDS uEye
UI-1240SE-C
x y z
O3D2xx MESA SR4000
vc,P/F – kv.vt/f = tx.√(z²P+y²P)
uc/F = ku.ut/F .
Other experiments with ToF cameras (1)
1. The ‘region of interest (ROI)’ is the floor ( … and not the garbage behind it ).
2. Since the camera is inclined (Rotx, Roty) one single ToF image
can fit the floor as:
z = a + b.x + c.y ( one image, CPUtime = 1 second ).
3. The surface of every strange object on the floor is covered by:
z < a + b.x + c.y
4. To become a stable black/white image a threshold is used:
bw = ( z + threshold < a + b.x + c.y ) . MESA SR4000
IDS uEye
UI-1240SE-C
Other experiments with ToF cameras (2)
Palette
Angles can be measured relatively
precise… use this fact if possible.
An equilibration of the angle errors is
also possible.
Automated ‘palette handling’..
using Time of Flight !
Desired accuracy:
1.5 cm absolute error on a
distance of 1.5 meter.
RGBd can be used as a
’ sub pixel GONIOMETER ! ‘
Muliti path !
Other experiments with ToF cameras (3)
ifm O3D2xx ToF 64 x 50 pixel 248 x 192
Inspect a random stack of ‘Palettes’
1 jL jM J2 jR J (Columns from 1:J)
D f
β0
β1 β2
tg(β0) = (jM-J2)/f tg(β0+β1) = (jL-J2)/f and tg(β0-β2) = (jR-J2)/f
β0 = atan2(jM-J2,f) β1 = atan2(jL-J2,f) - β0 and β2 = β0 + atan2(jR-J2,f)
cotg(α) = x/y = sin(β2–β1)/(2.sin(β1).sin(β2)) = tg(π/2-α)
x = D.cos(α) ; y = D.sin(α)
ToF searches at a lower resolution,
RGB is able to correct.
x
y
RGBd cameras
DepthSense 311
1. Find the z –discontinuity
2. Look for vertical and
forward oriented regions
3. Check the collineraity
4. Use geometrical laws
in order to find
x, y, z and b.
RGBd
Mesa SwissRanger SR4000
1. Remove bright pixels
2. Find the z –discontinuity
3. Look for vertical and
forward oriented regions
4. Check the collineraity
5. Use geometrical laws
in order to find
x, y, z and b.
ToF
ToF
IFM O3D2xx.
1. Remove weak defined
pixels.
2. Find the z –discontinuity
3. Look for vertical and
forward oriented regions
4. Check the collineraity
5. Use geometrical laws
in order to find
x, y, z and b.
% Program: 'CrossRatio_VR_diagonal_O2.m'.
3/5
Time of Flight cameras: vector interpretation
Reference =
TOF-source Measured
I0
I0 = b + A.ei.(φd + θ)
I1 = b + A.ei.(φd + θ + π/2)
I2 = b + A.ei.(φd + θ + π)
I3 = b + A.ei.(φd + θ + 3π/2)
b = (I0 + I1 + I2 + I3 )/4
A = (I0 + I1.e-iπ/2 + I2.e
-i.π + I3.e
-i.3π/2 )/4
φd + θ = arg(A)
I1
I2
I3
The calculation can easely be extended for:
- non-harmonic periodic illumination ( Fourier expansion !) ,
- more than 4 equidistant measurings in one cycle .
θ
φd
Remember: vector r.eiφ = r.[ cos(φ) + i.sin(φ) ]
r.cos(θ)
r.sin(θ)
b
top related