c o lo ur an algorithmic approach

Colouran algorithmic approach

Thomas Bangertthomas.bangert@qmul.ac.uk

http://www.eecs.qmul.ac.uk/~tb300/pub/PhD/ColourVision2.pptx

PhD Research Topic

understanding how natural visual systems process information

Visual system: • about 30% of cortex• most studied part of

brain• best understood part

of brain

Image sensors Binary sensor array

monochromatic ‘external retina’

Luminance sensor arraydichromatic colour

Multi-Spectral sensor arraytetrachromatic colour

What do these direct links to the brain do?

Lets hypothesise … When an astronomer looks at a star, how does he code the information his sensors produce?

It was noticed that parts of spectrum were missing.

Looking our own star – the sun

• x

Each atomic element absorbs at specific frequencies …

We can Code for these elements …

We can imagine how coding spectral element lines could be used for visual perception … by a creature very different to us… a creature which hunts by ‘tasting’ the light we reflect… seeing the stuff we are made of

Colour in this case means atomic structure and chemistry…

Where do we start with humans?

Any visual system starts with the sensor.

What kind of information do these sensors produce?How do we use that information to code what is relevant to us?

Let’s first look at sensors we ourselves have designed!

Sensors we buildX

Y

The Pixel

Sensors element may be:

Binary Luminance RGB

The fundamental unit of information!

The Bitmap

2-d space

represented by integer array

0 1 2

0

1

What information is produced?

2-d array of pixels:

Black & White Pixel:– single luminance

value, usually 8 bit Colour Pixel

– 3 colour values, usually 8-bit RGB

What does RGB mean?• It is an instruction for producing

light stimuli• Light stimuli for a human

standard observer• Light stimuli produce

perception• RGB codes the re-production of

measured perceptual stimuli• It is assumed that humans are

trichromatic• It tells us nothing about what

colour means!

The Standard Observer

CIE1931 xy chromaticity diagramprimaries at: 435.8nm, 546.1nm, 700nm

The XYZ sensor response

xxx y z

now we extract the colour information from the sensor readings

The Math:

yyx y z

… 2-d as z is redundant

Understanding CIE chromaticity

White in center

Saturated / monochromatic colours on the periphery

Best understood as a failed colour circle

Everything in between is a mix of white and the colour

xxx y z

yyx y z

Does it match?The problem of

‘negative primaries’

But does it blend?

Monochromatic Colours

What the Human Visual System (HVS) does is very different!

?

Human Visual

System (HVS)

Part 1

Coding Colour

The Sensor2 systems: day-sensor & night-sensor

To simplify: we ignore night sensor system

Cone Sensors very similar to RGB sensors we design for cameras

BUT: sensor array is not ordered

arrangement is random

note:very few blue sensors, none in the centre

sensor pre-processing circuitry

First Question: What information is sent from sensor array

to visual system?

Very clear division between sensor & pre-processing (Front of Brain) andvisual system (Back of Brain) connected with very limited communication link

Receptive Fields

All sensors in the retina are organized into receptive fields

Two types of receptive field.Why?

What does a receptive field look like?

In the central fovea it is simply a pair of sensors.

Always 2 types:• plus-centre• minus-centre

What do retinal receptive fields do?

Produce an opponent value:simply the difference between 2 sensors

This means:it is a relative measure, not an absolute measure

andno difference = no information to brain

Sensor Input

Luminance Levels

it is usual to code 256 levels of luminanceLinear: YLogarithmic: Y’

Receptive Field Function

- - -- - -- - -

+ + ++ + ++ + +

- - -- - -- - -

- - -- - -- - -

- - -- - -- - -

- - -- - -- - -

- - -- - -- - -

- - -- - -- - -

- - -- - -- - -

+ + ++ + ++ + +

- - -- - -- - -

+ + ++ + ++ + +

+ + ++ + ++ + +

+ + ++ + ++ + +

+ + ++ + ++ + +

+ + ++ + ++ + +

+ + ++ + ++ + +

+ + ++ + ++ + +

Min Zone

Max-Min Function

Output is difference between average of center and max/min of surround Max

Zone

Tip of Triangle

Dual Response to gradients

Why?

Often described assecond derivative/zero crossing

AbstractedNeurons only produce positive values.Dual +/- produces positive & negative values.

Together: called a channel means signed values.

Produces directional information

+-

+-

+-

+-

+-+- +-

+-Location, angle luminance, equiluminance and colour

Information sent to higher visual processing areas

This is a sparse representation

From this the percept is created

This is a type of data compression.Only essential information is sent!

Conversion from this format to bitmap?

starting with the sensor:Human Sensor Response

to non-chromatic light stimuli

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rptio

n (%

)

RGB

HVS Luminance Sensor IdealizedSe

nsor

Valu

e

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λδλ−

A linear response in relation to wavelength.Under ideal conditions can be used to measure wavelength.

Spatially Opponent

HVS:Luminance is always measured by taking the difference between two sensor values.Produces: contrast value

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λδλ−

Sens

or V

alue

Wavelength(λ)λ

0.8

0.6

0.2

0.0

1.0

0.4

λδλ−

Which is done twice, to get a signed contrast value

Moving from Luminance to Colour

• Primitive visual systems were in b&w• Night-vision remains b&w

• Evolutionary Path– Monochromacy– Dichromacy (most mammals – eg. the

dog)– Trichromacy (birds, apes, some monkeys)

• Vital for evolution: backwards compatibility

Electro-Magnetic Spectrum

Visible Spectrum

Visual system must represent light stimuli within this zone.

Colour Vision Young-Helmholtz

Theory

Argument:Sensors are RGB thereforeBrain is RGB

3 colour model

Hering colour opponency model

Fact: we never see reddish green or yellowish blue.

Therefore:colours must be arranged in opponent pairs:

RedGreenBlueYellow

4 colour model

Colour Sensorresponse to monochromatic light

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rptio

n (%

)

RGB

370 nm 445 nm 508 nm 565 nm

700 nm330 nm 400 nm 500 nm 600 nm

1.0

0.5

0.0

Human

Bird

4 sensorsEquidistant on spectrum

How to calculate spectral frequency with 2 poor quality luminance sensors.

 Roughly speaking:

Sensor Value

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-Δ λ λ+Δ

RG

a shift of Δfrom a known reference point

the ideal light stimulusSe

nsor

Val

ue

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-δ λ λ+δ

RG Monochromatic Light

Allows frequency to be measured in relation to reference.

Problem:natural light is not ideal

Sens

or V

alue

Wavelength

0.8

0.6

0.2

0.0

1.0

0.4

λ-δ λ λ+δ

RG

• Light stimulus might not activate reference sensor fully.

• Light stimulus might not be fully monochromatic.

ie. there might be white mixed in

Sens

or V

alue

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

Solution:

A 3rd sensor is used to measure equiluminance.

Which is subtracted.

Then reference sensor can be normalized

Equiluminance & Normalization

Also called Saturation and Lightness.

• Must be removed first – before opponent values calculated.

• Then opponent value = spectral frequency

• Values must be preserved – otherwise information is lost.

a 4 sensor designSe

nsor

Val

ue

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

2 opponent pairs• only 1 of each pair can be active• min sensor is equiluminance

,R G B y

What is Colour?Colour is calculated exactly the same as luminance contrast. The only difference is spectral range of sensors is modified.Colour channels are: RGBy

Uncorrected colour values are contrast values.But with white subtracted and normalized:Colour is Wavelength!

How many sensors?

4 primary colours require 4 sensors!

Human Retina only has 3 sensors!What to do?

We add an emulation layer.

Hardware has 3 physical sensorsbut emulates 4 sensors

Wavelength (nm)

Sens

or R

espon

se

460 580 640520 550 610490430 670400370 700

0.8

0.6

0.2

0.0

1.0

1.2

1.4

0.4

R G B

No maths … just a diagram!

Testing Colour Opponent model

What we should see

What we do see

Unfortunately it does not matchThere is Red in our Blue

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rptio

n (%

)RGB

Pigment Absorption Data of human cone sensors

350 400 450 500 550 600 650 7000

10

20

30

40

50

60

70

80

90

100

Wavelength (nm)

Abso

rptio

n (%

)

RGB

Red > Green

Solution:HVS colour representation must be circular!

Which is not a new idea, but not currently in fashion.

540nm

620nm

480nm

Dual Opponency with Circularity

an ideal model using 2 sensor pairs

Sens

or V

alue

Wavelength(λ, in nm)400300 430 460 490 520 550 580 610 640 670 700

0.8

0.6

0.2

0.0

1.0

0.4

… requires 2 independent channels which give 4 primary colours

Yellow added as a primary!

Which allows a simple transform to circular representation

180°0°

90°45°

135°

315°

270°

225°

Opponent Values HueA simple transform from 2 opponent values to a single hue value

How might HVS do this?we keep 2 colour channels but link them

180°0°

90°45°

135°

315°

270°

225°

0

+64

+127

R = 255 G = 0 B = 128

Y - +

Ψ B

CG R

180°0°

90°45°

135°

315°

270°

225°

Travelling the Colour Wheel (Hue)

One Chroma channel is always at max or minThe other Chroma channel is incremented or decremented

Rules:if (CB==Max)CR--if (CR==Max)CB++if (CR==Min)CB--if (CB==Min)CR++

180°0°

90°45°

135°

315°

270°

225°

0

-127

+128

R = 255 G = 128 B = 0

Y - +

CB Ψ B

CR G R

0

-127

0

R = 255 G = 255 B = 0

Y - +

CB Ψ B

CR G R

+ -

Colour Wheel

Simple rule based system that cycles through the colour wheelAllows arithmetic operations on colour

180°0°

90°45°

135°

315°

270°

225°

0

0

+127

R = 255 G = 0 B = 0

Y - +

CB Ψ B

CR G R

0

+64

+127

R = 255 G = 0 B = 128

Y - +

CB Ψ B

CR G R

0

+127

+127

R = 255 G = 0 B = 255

Y - +

CB Ψ B

CR G R

0

+127

+64

R = 128 G = 0 B = 255

Y - +

CB Ψ B

CR G R

0

+127

0

R = 0 G = 0 B = 255

Y - +

CB Ψ B

CR G R

0

+127

-64

R = 0 G = 128 B = 255

Y - +

CB Ψ B

CR G R

0

+127

-127

R = 0 G = 255 B = 255

Y - +

CB Ψ B

CR G R

0

+127

-127

R = 0 G = 255 B = 0

Y - +

CB Ψ B

CR G R

0

+127

-127

R = 128 G = 255 B = 0

Y - +

CB Ψ B

CR G R

0

-127

0

R = 255 G = 255 B = 0

Y - +

CB Ψ B

CR G R

0

-127

+128

R = 255 G = 128 B = 0

Y - +

CB Ψ B

CR G R

0

-64

+127

R = 255 G = 128 B = 0

Y - +

CB Ψ B

CR G R

0

0

+127

R = 255 G = 0 B = 0

Y - +

CB Ψ B

CR G R

Part 2Accurate Colour reproduction

First problem:

Real world is not monochromatic

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

350

367

383

400

416

433

449

466

482

499

515

532

548

565

581

598

614

631

647

664

681

697

714

730

Spectrum of a common yellow flower

Accurate Colour reproduction

second problem:human colour vision is

inaccurateprone to ‘making stuff up’

varies from person to person

The closer the sensors the less accurate the color information.

All humans are to an extent color blind … compared to animals like birds.

Examples of real world colour?

Colours are often computed, not measured!

… an extreme example

What is the colour?

Accurate colour reproduction … for dual channel opponency

Problem # 1

very easy to solve

we simply assume monochromacywhen stimuli are not monochromatic opponent channels simply subtract to 0

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

350

367

383

400

416

433

449

466

482

499

515

532

548

565

581

598

614

631

647

664

681

697

714

730

green, yellow and red are activer-g = 0b = 0

leaving only yellowstimuli equivalent to monochromatic

Accurate colour reproduction … with primaries

Only primaries are true coloursall other colours are intermediary

… and can be generated by proportions of primaries!

Accurate colour reproduction … for humans

Any colour may be displayed by a combination of 2 primaries

but the location of primaries can vary between individualsand intermediary locations can be distorted

Problem # 2

Solution toAccurate colour reproduction …

for the individual human

1. primaries must be mapped for the individual

2. mid-points must be mapped

467 517 573 644503 603

545

Provides an individual colour profile … a map of the primaries and intermediary points. Can be repeated recursively for greater precision.

Does it work?

• Colour opponency requires primaries to be precisely located.

• For humans this would be virtual primaries• Is there evidence that this is so?• Do the colours match?

This will be tested empirically …

• monochromator

• Xenon light sourceequal light across visible spectrum

Apparatus

Procedure• generate subject selectable

monochromatic stimuli• subject selects colour• virtual primaries are

calculated

Preliminary resultsBlue Green Yellow Red

455 520 580 650465 520 580 640465 520 575 640470 525 585 655460 520 575 640465 520 575 650465 525 575 635475 515 570 640470 522.5 570 635

482.5 525 572.5 635462.5 532.5 580 640

471.25 526 578.3 647467.2 522.6 576.3 642.3

7.19 4.5 4.5 6.7AverageStd Dev

Discussion

small sample with inaccurate toolsbut

• primaries appear to be are very closely grouped and spaced equidistantly – exactly as predicted

In Future

• visits to optometrist will include a colour test

• Colour displays may be set by colour ‘prescription’

http://www.eecs.qmul.ac.uk/~tb300/pub/PhD/ColourVision2.pptx

ReferencesPoynton, C. A. (1995). “Poynton’s Color FAQ”, electronic preprint.http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html

Bangert, Thomas (2008). “TriangleVision: A Toy Visual System”, ICANN 2008.

Goldsmith, Timothy H. (July 2006). “What birds see”. Scientific American: 69–75.

Neitz, Jay; Neitz, Maureen. (August 2008). “Colour Vision: The Wonder of Hue”. Current Biology 18(16): R700-r702.

Questions?