Transcript
Page 1: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

視覚情報処理論

(Visual Information Processing )開講所属: 学際情報学府

水(Wed)5 [16:50-18:35]

Page 2: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Schedule• 9/ 26 Introduction (Prof. Oishi)• 10/3 Patch-based Object Recognition (1) (Dr. Kagesawa)• 10/10 Patch-based Object Recognition (2) (Dr. Kagesawa)• 10/17 Computer Vision basics (1)(Prof. Oishi)• 10/24 Computer Vision basics (2)(Prof. Oishi)• 10/31 Image and Video Inpainting (1) (Dr. Roxas) (※in English)• 11/7 Image and Video Inpainting (2) (Dr. Roxas) (※in English)• 11/14 (Cancelled)• 11/21 Vision for Robotics Applications (1) (Dr. Sato)• 11/28 Vision for Robotics Applications (2) (Dr. Sato)• 12/5 3D Data Visualization (1) (Dr. Okamoto)• 12/12 3D Data Visualization (2) (Dr. Okamoto)• 12/19 3D Data Processing (1) (Prof. Oishi)• 1/9 3D Data Processing (2) (Prof. Oishi)

Page 3: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Computer Vision Paradigm (Marr)

2.5D Image

2D Image

3D representation

Integration

Brightness Texture Line drawing Stereo Motion

Observer oriented

3D Feature Extraction(shape-from-x)

Object oriented 3D Model

Page 4: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Digital image processing (2D)

Page 5: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

What is digital image?Analog information (Film, Painting, Real world)

Digital image• Digital camera• Smart phone• PC data, IT• Digital broadband

Discretization & Sampling

Page 6: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

SamplingDiscrete segmentation of analog data

Analog data(Time and value are sequential)

Sampling data(Time is discrete)

Sampling interval

Page 7: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Sampling2D digital image

Image resolution is defined by sampling interval

Page 8: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

What is pixel?Unit of 2D digital image Space sampling

0 1 N-1

0

1

M-1

columns

rows

Digital imageM x N pixels

n

m

Page 9: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Sampling-Resolution

320 x 240pixels

160 x 120pixels

80 x 60pixels

40 x 30pixels

Page 10: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

QuantizationSampled values are discretized

Sampled data(Time line is discrete)

Quantization bit:3 bit = 8 level8 bit = 256 level

Digital data(Both time and value are discrete)

Page 11: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Quantization2-D digital image

Number of color depends on quantization bit

0

0

0

1

1

1

1

0

0

1

2

2

2

1

0

0

2

3

3

2

1

0

2

3

5

3

2

0

0

2

3

3

3

2

0

0

1

2

2

2

0

0

0

1

1

1

0

0

0

Color is represented by number

Page 12: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color representationHow many colors do we need?

4colors(2bi)

16colors(4bit)

256colors(8bit)

16.7 millioncolors(32bit)

Page 13: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

High Dynamic Range Imaging: HDRI

Page 14: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Exposure time - Intensity [Mathias Eitz, Claudia Stripf,High Dynamic Range Imaging, 2007]

Under Exposure Over Exposure

Page 15: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Dynamic range

Human

Camera

Page 16: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Multiple capturing

Page 17: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Camera response function

Exposure Exposure

Page 18: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Estimation of camera response functionCapturing multiple images with different exposure time

Page 19: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Computation of response curve

Zij = f (Eitj )f −1 (Zij ) = Eitj

ln f −1 (Zij ) = ln Ei + lntj

Log Exposure

Zij : Pixel valuef : Camera response functionEi : Radiancetj : Exposure time

Page 20: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Displaying HDRI

HDRI

LDRI

Page 21: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Tone mappingLinear mapping Logarithmic mappingGlobal Reinhard operator

L (x, y) = L(x,y) / 1+L(x,y)

Page 22: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Results of tone mapping

without tone mapping with tone mapping

Page 23: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

HDRI Video [Kalantari et al. Patch-Based High Dynamic Range Video, TOG 2013]

Page 24: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D
Page 25: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Filtering

Page 26: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

FilteringPre-processing for Computer Vision

• Noise reduction• Image enhancement• Feature extraction

FILTER ?

Page 27: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Spatial – Frequency filterProcessing in spatial domain

• Neighboring pixels

Processing in frequency domain• Using Fourier Transform

Page 28: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Image NoiseNoise source

• Capturing

• Compression/Transfer

Page 29: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Mean filterReplace value with mean of neighboring points

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5 3 x 3(5 x 5)(7 x 7)

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

1 / 9

10 / 9

8 / 9

8 / 9

8 / 9

5 / 9

0 / 9

9 / 9

7 / 9

7 / 9

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

7

8

Page 30: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Mean filterWeighted average

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

4 / 16

2 / 16

2 / 16

2 / 16

1 / 16

1 / 16

2 / 16

1 / 16

1 / 16

40 / 16

16 / 16

16 / 16

16 / 16

5 / 16

0 / 16

18 / 16

7 / 16

7 / 16

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

8

6

Page 31: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Mean filter (Smoothing, Averaging)for Gaussian noise

Noise image(5% Gaussian)

Average Weighted average

Page 32: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Mean filterEx. Shot noise

Noise image(Random binary)

Average Weighted average

Page 33: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Non-linear filterMaximum filter

• Replace target value with maximum value in a window

Minimum filter• Replace target value with minimum value in a window

Median filter

Page 34: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

1098887750

7859108780

ソート 中央値

Median filterReplace target value with median value in a window

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

0 5

5

3

3

1

4

10

8

8

7

6

8

5

0

9

4

8

5

9

10

9

7

7

5

8

8

Page 35: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Median filter3 x 3 Filter

Shot noiseGaussian noise

Page 36: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Edge detection

Page 37: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Edge typesStep edge

Roof edge

Peak edge

x

x

x

Page 38: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

1-D edge differentialFirst and second order differentials

Fig. from Digital Image Processing (Springer)

Original signal

First order

Second order

Page 39: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Gradient-baseOperator of first order differential

Discrete difference equation

y

f

x

fyxf ,,

nmfnmfnmf

nmfnmfnmf

y

x

,1,,

,,1,

2 x 2 size

1,1,,

,1,1,

nmfnmfnmf

nmfnmfnmf

y

x

3 x 3 size

Strength and direction of edge

Page 40: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Gradient-baseOperators

• Roberts

• Prewitt

• Sobel

10

01

01

10\/ DD

111

000

111

101

101

101

yx DD

121

000

121

101

202

101

yx DD

Page 41: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Gradient-basePrewitt operator

Dx Dy

Page 42: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Laplacian operatorOperator of second order differential

Strength of edge is estimated

010

141

010

2

2

1

121222yx DD xxx DDD

2

010

141

0102

111

181

1112

4 direction 8 direction

yyy DDD

2

Page 43: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Laplacian operatorLaplacian operator

4 direction 8 direction

Page 44: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Laplacian of GaussianDifferential operation is weak to noiseGaussian filter (noise reduction) -> Laplacian operator

Laplacian of Gaussian 222 2/

2

1,G

yxeyx

222 2/2

22

42 2

2

1,G

yxeyx

yx

Page 45: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Laplacian of GaussianLOGオペレータ

1 2

Page 46: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Line drawing analysis

Page 47: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Line drawing extraction

Original image Differential image Line drawing image

Page 48: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

3D Information form Line DrawingGiven

• Line drawing(2D)Find

• 3D object that projects to given lines

Find• How do you think it’s a cube,

not a painted pancake?

Page 49: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Line types

convex concave

occluding occluding

Page 50: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Labeling a Line Drawing

Easy to label lines for this solid→Now invert this in order to understand shape

Page 51: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Enumerating Possible Line Labeling without Constraints

•9 lines•4 labels each

→4x4x4x4x4x4x4x4x4= 262,144 possibilitiesWe want just one reality

must reduce surplus possibilities→Need constraints (by 3D relationship)

Page 52: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Huffman & Clows Junction DictionaryAny other arrangements

cannot ariseHave reduced configuration

from 208 to 12

• L-type - 6

• ARROW-type - 3

• FORK-type - 3

Page 53: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Constraints on LabelingWithout constraints-- 262,144 possibilitiesConsider →3x3x3x6x6x6x3= 17496 possibilitiesconstraints

We can reduce more bycoherency/consistency along line.

Page 54: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Labeling by Constraint Propagation“Waltz filtering”By coherence rule, line label constrains neighborsPropagate constraint through common vertexUsually begin on boundaryMay need to backtrack

Page 55: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Impossible objectsNo consistent labelingBut some do have a consistent labeling

• What’s wrong here?

Page 56: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Limitations of Line LabelingOnly qualitative; only gets topologySomething wrong

Page 57: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color theory

Page 58: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color Theory for Computer VisionColor in several domains:

• Physics• Human vision• Psychophysics• Perception• Computer Vision

Color problems in Computer Vision:• Color for segmentation• Color for reflection physics

Page 59: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color spectrum

Intensity at each wavelength

Page 60: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

RGB imageRGB color model

r=255g=5b=10

DSC(Digital Still Camera)

Spectrum is compressed to three color valuesResponse function

Page 61: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

IlluminationSpectrum is richer than RGB

Page 62: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Are RGB enough?5900K light

MetamerismNatrium light

Standard illumination

D50 light

Page 63: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Spectral distribution measurement

Page 64: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Interference CameraSpectrum varies along the position

Interference filter

Page 65: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Y

Page 66: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Panoramic Multispectral Imaging SystemLCTF Capturing System

Automatic Pan/Tilt Platform

Page 67: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

LCTF Capturing system

・・・

t (s)400nm ~ ~720nm

Target scene

LCTFMonochromatic CCD camera

400nm404nm408nm416nm・・・nm・・・nm・・・nm・・・nm・・・nm・・・nm・・・nm712nm716nm720nm

PC

[Tominaga et al. 00]

Page 68: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Tumulus and hill

Page 69: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

In what condition painted?under sun-lightunder torch?

� U Tokyo / Topan / Kyushu National Museum

Page 70: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Simulation ResultsSimulation results suggest that

• Painted most likely under sun light• First paints, and then covers the tumulus

Torch Sun light

Page 71: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Point light source(Incandescent)

Spectral measurement sensor

Target object: Tomato

RGB camera

Data analysis

Spectral measurement of Aging process

Page 72: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Measurement time: every 12 hours in 14 days

0

0.05

0.1

0.15

0.2

0.25

380 480 580 680

波長(nm)

分光反

射率

Temporal variation

Page 73: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

1st principal component proportion : 61.1%Regressioncurve: -0.1996240.0153333t

3t0.00013939358t0.00000773 231

Y

2nd principal component proportion : 23.3%Regressioncurve: 0.008506940.0720887t0.0225962t

0.002247t86t0.00008970-553t0.000001252

3452

Y

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

380 480 580 680

波長(nm)

主成分の係数

第一主成分

第二主成分

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0 5 10 15 20 25

日数

得点

第一主成分得点

第二主成分得点

第一主成分得点(回帰曲線)

第二主成分得点(回帰曲線)

Principal component analysis

Page 74: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Reflectance image

Color image Texture image Reconstructed image

3D model rendering

Page 75: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Human visionRetina

Retina has 4 cells

•“red” cone cell•“green” cone cell•“blue” cone cell• Rod cell Intensity

Color

Page 76: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Human vision

380nm 760nm

)(bC

)(gC

)(rC

Response of red cone =

Response of green cone =

Response of blue cone =

dECr )()(760

380 dECg )()( dECb )()(

Page 77: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color spacered = green =blue =

dECr )()( dECg )()( dECb )()(

If we approximate spectral power distribution by vector, it’s a matrix multiplication.

)(E

red greenblue

=)(rC)(gC)(bC )(E

13 3 1

spectral space : infinitely many dimensionscolor space : 3 dimensions

Page 78: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Alternate color spaceother isomorphic color spaces formed by linear transforms

red greenblue

=)(rC)(gC)(bC )(E

define new axesABC

=red greenblue

=

=

)(rC)(gC)(bC )(E

)(a)(b)(c )(E

linear transform gives new axes

new response function

green

bluered

A

C B

Page 79: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Psychophysical color (X-Y-Z)international standard color space agreed upon byCommision Internationale de I’Eclairage (CIE)• particular linear transform of human cone responses• Two spectral distributions that result in the same values in the

space appear indistinguishable • all colors have positive x, y, zEach point in X-Y-Z is a different colorChromaticity

x = X / (X+Y+Z) ≒ R / (R+G+B)y = Y / (X+Y+Z) ≒ G / (R+G+B)z = Z / (X+Y+Z) ≒ B / (R+G+B)since x+y+z = 1, z = 1-(x+y). --- redundant usually plotted o x-y diagram

Each point is many XYZ colors

Page 80: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Chromaticity diagram

r = R / (R+G+B)g = G / (R+G+B)b = B / (R+G+B)

Page 81: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color perceptionHow do people describe color ?NOT “X-Y-Z” nor “R-G-B” !People use cylindrical coordinates.hue, saturation, brightness

B H

S

blue

white

violetred

yellow

green

SH

One plane of constant brightness

hue+saturation form polar coordinates

relationship to red-green-blue

Page 82: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Hue-Saturation-Brightness (Value) Space

blue

black whitehue

Page 83: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Photometric properties

Page 84: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

)()()( ESI

Observed color

ObservedSurfacereflectance

Illumination

Page 85: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Role of Color in Robot Vision1. Feature space for 2D segmentation

more features → be er discrimina�on2. Color physics of reflection

What physical information can color provide?

Page 86: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color reflection physicssurface reflection and body reflection

bodyair

incident lightsurfacereflection

bodyreflection

internalpigment

Page 87: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Separating reflection components by colorPixel color vectors are

Make a histogram fit parallelogramProject each pixel onto vectorsDetermine everywhere

Klinker 88bbss CC

bs CC ,

bs ,

body reflection

surface reflection

b

ssC

bC

R

G

B

Page 88: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Color space analysis

dbLL

dgLL

drLL

dbL

dgL

drL

B

G

R

C

bs

bs

bs

)())()((

)())()((

)())()((

)()(

)()(

)()(

bbss

b

b

b

b

s

s

s

sbs

b

b

b

s

s

s

bs

bs

bs

CC

B

G

R

B

G

R

dbO

dgO

drO

dbI

dgI

drI

dbO

dgO

drO

dbI

dgI

drI

dbOI

dgOI

drOI

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)()(

)())()((

)())()((

)())()((

body color vector in RGB spacesurface color vector in RGB spaceColor vector at a pixel is a linear combination of surface + body reflection color vector

Page 89: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Dichromatic Reflection Modelsurface reflection has SPD of incident lightbody reflection has SPD of body color

brightness reflected)( L

surface reflection body reflection

SPD of body colorSPD of incident light

Page 90: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Klinker et al.’s method

Steps:

1. Color segmentation

2. T-shape identification

Page 91: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Separation Results

Page 92: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Chromaticity-Intensity Space

a. Specular image c. Chromaticity Intensity space

a b c

b. Spatial Intensity space

Page 93: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

96

Iteration Framework

Page 94: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Result: a single object

Input image Specular-free image

Page 95: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Separation Result

Diffuse reflection component

Specular reflection component

Page 96: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Separation using High Frequency Illumination

[S.K. Nayar et al. SIGGRAPH 2006]

Page 97: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D
Page 98: 視覚情報処理論 (Visual Information Processing )7 5 8 8 Median filter 3 x 3 Filter Gaussian noise Shot noise Edge detection Edge types Step edge Roof edge Peak edge x x x 1-D

Summary2D digital image processingEdge detectionLine drawing analysisColor theoryPhotometric properties


Top Related