Romain BretteEcole Normale Supérieure, Paris

romain.brette@ens.fr

Computing with neural synchrony:an ecological approach to neural computation

Perception as pattern recognition

Marr (1982). Vision. Freeman & Co Ltd

The main function of sensory systems is often described as pattern recognition.

Sensory input« image »

Neural representation« features »

Perception

Pattern recognition

Memory

This is the main paradigm in standard neural network theory (e.g. perceptron).

One major issue of perceptual systems is invariance: the same object can correspond to many different patterns. How do you build a system that is invariant to these changes?

The ecological approach“Ask not what’s inside your head but what your head is inside of”

James J Gibson

James J Gibson (1979). The ecological approach to visual perception.JK O’Regan & A. Noë. A sensorimotor account of vision and visual consciousness.

Main point: sensory signals come from real things in the world, and therefore are tightly constrained by the laws of physics. Perceiving is about grasping sensory laws, the “invariant structure” of sensory signals (or sensorimotor laws).

Example, pitch:

This acoustical wave elicits a pitch percept (musical note)

This other wave elicits the same pitch (same note), but the pattern is different.

However, the structure is the same (=periodicity)

Pattern vs. structurePattern recognition

This is an A4

This is another instance of an A4

I don’t know!

What is this signal?

→ label this pattern as A4

I don’t know!

→ label this pattern as A4

Structure identification

This is an A4

I don’t know! But I notice that S(t+T)=S(t)

→ label this structure as A4

S(t)

T

Invariance is learned with many examples Invariance is intrinsic, one example is enough

I notice that S(t+T)=S(t)

It’s an A4!

Olfaction(This is just meant as a pedagogical example)

fluctuates (air turbulences)

C1(t) = a1 x [O](t)

C2(t) = a2 x [O](t)

C3(t) = a3 x [O](t)

C4(t) = a4 x [O](t)

affinity of receptor 1 to the odor

Odor identity is defined by receptor affinities

Sensory laws or « invariant structure »:

C1(t)/ C2(t) = a1/a2

a2.C1(t) = a1.C2(t)

constant ratio between signals

two views of the signals are identical

NEURAL SYNCHRONY AND SENSORY STRUCTURE

Encoding signals into spikesZ

. M

ain

en,

T. S

ejn

ow

ski, Science

(1

99

5)

Spike timing is reproducible in vitro for time-varying inputs

Integrate-and-fire model:

Encoding signals into spikesIntegrate-and-fire model:

Main points:

1) Temporal precision is determined by intrinsic noise, rather than signal fluctuation timescale.

2) Neurons are precise in the fluctuation-driven regime (mean input below threshold)

The synchrony receptive field

A

B

« Synchrony receptive field » = set of stimuli S making A and B fire synchronously

= {S | NA(S) = NB(S)}

a law followed by sensory signals Sor « invariant structure »

Structure and synchronySynchrony patterns reflect invariant sensory structure

X S

Structured signals:

source of variation

sensory signals

invariant transformation

T

X

S

pitch

Computing with synchrony

no response

A

B

Synchrony receptive field = {S | NA(S) = NB(S)}

C

Neuron C responds to coincidences between A and B= when S is in SRF(A,B)

Condition for coincidence detection: fluctuation-driven regime

THE OLFACTORY EXAMPLE

Olfactory receptors

fluctuates (air turbulences)

C1(t) = a1 x [O](t)

C2(t) = a2 x [O](t)

C3(t) = a3 x [O](t)

C4(t) = a4 x [O](t)

A

B

C

receptor neurons with different sensitivities

sA x a1 x [O](t)

sB x a1 x [O](t)

sC x a4 x [O](t)

s x aA and C synchronize for some odor (sA x a1 = sC x a4)

B and C synchronize for another odor (sB x a1 = sC x a4)

Spiking neuron model

Neurons with different receptor types (a) and sensitivities (s)

color = a x s for odor A

One neural assembly for odor A.

Receptors with the same color (=synchronous for odor A) project to the same neuron in the assembly.

Spiking neuron model

color = a x s for odor Bsame neurons different « synchrony groups »

Another neural assembly for odor B.

Receptors with the same color (=synchronous for odor B) project to the same neuron in the assembly.

Spiking neuron model

odorconcentration

Neurons wired to detect synchrony patterns produced by odor A fire.

Spiking neuron model

odorconcentration

Neurons wired to detect synchrony patterns produced by odor B fire.

Spiking neuron model

odorconcentration

Neurons from group B still fire, neurons from group A don’t.

Receptor neurons start saturating.

Spiking neuron model

odorconcentration

Distracting odor reduces firing in group A, but doesn’t increase firing in group B

Spiking neuron model

odorconcentration

LEARNING SENSORY STRUCTURE

The problem

1) We want the network to learn the correct mapping

i.e., connecting neurons that are simultaneously active for a specific structure to a postsynaptic neuron

sounds like STDP!

2) We want neurons to respond only to coincidences

Coincidence detection and homeostasis

• A coincidence detector must only fire to coincidences, i.e., rarely

• Homeostatic mechanism: enforce a target firing rate F• Example, synaptic scaling:

w→(1-a)w when the neuron spikes

dw/dt = b.w otherwise

Equilibrium: F=b/a

Weight change = -a.w.F.dt +b.w.dt

This homeostastic mechanism does not change relative weights

Learning structure with STDPSTDP

(only potentiation)

+ synaptic scaling

pre

pre

pre

post

Potentiation of coincident inputs

Learning to identify odorsRandom presentation of odors A and B, fixed concentration

At the end: neurons are tuned to either A or B

Learning to identify odorsConcentration-invariant responses:

At the end: neurons are tuned to either A or B

Conc

entr

ation

BINAURAL HEARING

Binaural hearing without diffraction

Synchrony when:S(t-dR-δR)=S(t-dL-δL)

dR-dL = δR +δL

Independent of source signal

But this is simplistic!

S(t-dR) S(t-dL)

S(t)

This is essentially the Jeffress model of sound localization

(invariant structure)

Binaural hearing in real lifeFR,FL = location-dependent acoustical filters(HRTFs/HRIRs)

Delay:

low frequency high frequency

Sound propagation is more complex than pure delays!

Binaural hearing in real lifeFR,FL = location-dependent acoustical filters(HRTFs/HRIRs)

Delay:

low frequency high frequency

ITDs:

FRONT BACK

Frequency

ITD

(ms)

Binaural structure and synchrony receptive fields

FR,FL = HRTFs/HRIRs (location-dependent)

NA, NB = neural filters(e.g. basilar membrane filtering)

input to neuron A: NA*FR*S (convolution)input to neuron B: NB*FL*S

Synchrony when: NA*FR = NB*FL

SRF(A,B) = set of filter pairs (FL,FR)= set of source locations= spatial receptive field

Independent of source signal S

The hypothesis

FR*SFL*S

NA*FR*SNB*FL*S

Each binaural neuron encodes an element of binaural structure

Proof of concept

FR

FL

γi

γi

GRj

GLj

Sounds: noise, musical instruments, voice (VCV)

Acoustical filtering: measured human HRTFs

Gammatone filterbank +more filters Spiking: noisy IF

models

Coincidence detection: noisy IF models

Goodman DF and R Brette (2010). Spike-timing-based computation in sound localization. PLoS Comp Biol 6(11): e1000993. doi:10.1371/journal.pcbi.1000993.

Activation of all assemblies as a function of preferred location:

Experimental prediction

Cells (cat IC)

HRTFs

Cells (IC)

HRTFs

Best phase of a neuron vs. frequency= Interaural phase difference vs. frequency for preferred source location PUT A

CELLCP

CD

Best

pha

se

Input frequency (Hz)

MORE EXAMPLES

Visual edges

SRF(A,B) = translation-invariant image

In LGN: correlation is tuned to orientation!

Stanley et al. (2012). Visual Orientation and Directional Selectivity through Thalamic Synchrony. J Neurosci

LGN

Visual edges

« It looks like a Gabor wavelet » « It is translation-invariant »

Cross-correlation vs. autocorrelation

OK but what’s the difference with the standard V1 model?

Barlow & Berry (2010). Cross- and auto-correlation in early vision. Proc Royal Soc B.

Pattern recognition Structure detection

Impact sounds

• Decay time indicates material (metal/wood)• Resonant frequencies indicate shape• Amplitude of modes are linked to impact properties

Take home messages

• Synchrony reflects sensory structure• STDP learns structure• Computing with neural synchrony is detecting

structure or « laws » in sensory signals≠ pattern recognition

• Invariance is not an issue anymore because structure is an invariant

Thank you

Bertrand Fontaine

Cyrille Rossant

Dan Goodman

Philip Joris (Leuven)

Victor Benichoux

Anna Magnusson(Stockholm)

Marc Rébillat JonathanLaudanski

Jose Peña (NY)

Impact of reflections on binaural cues

Boris Gourévitch

Coincidence sensitivity of neuronsModel fitting

HRTF measurements and analysisAnalysis of in vivo recordingsBinaural model

Binaural model (« proof of principle »)Brian simulator

Auditory models(« Brian Hears »)

Analysis of HRTFsPitch modelBinaural model

In vivo electrophysio(cats)

In vivo electrophysio(owls)

In vitro electrophysiology

And: Makoto Otani (BEM simulations)Renaud Keriven (3D models)

romain.brette@ens.frhttp://audition.ens.fr/brette/

Publications on synchrony-based computing

• Reliability of spike timing in models: Brette, R. and E. Guigon (2003). Reliability of spike timing is a general property of spiking model neurons. Neural Comput 15(2): 279-308.• Coincidence detection: Rossant C, Leijon S, Magnusson AK, Brette R (2011). Sensitivity of noisy neurons to coincident inputs. J Neurosci 31(47):17193-17206.• Computing with synchrony: Brette R (2012). Computing with neural synchrony. PLoS Comp Biol• Sound localization with binaural synchrony: Goodman DF and R Brette (2010). Spike-timing-based computation in sound localization. PLoS Comp Biol 6(11): e1000993. doi:10.1371/journal.pcbi.1000993.• Simulation: Goodman, D. and R. Brette (2009). The Brian simulator. Front Neurosci doi:10.3389/neuro.01.026.2009.

Invariant structure in perception (psychology): James J Gibson (1979), The ecological approach to visual perception. Boston: Houghton Mifflin.

A FEW WORDS ABOUT COINCIDENCE DETECTION

An example

event event100ms

20mV

2Hz

8Hz

4000 independent Poisson excitatory inputs + 1000 inhibitory inputs

synchrony events involving 10 random synapses, at rate 40 Hz

Pairwise correlation: 0.0002 (probably not experimentally detectable!)

Why tiny correlations may have large postsynaptic effects

• In a balanced regime, the output rate depends on both the mean and the variance of the input

• Consider N random variables Xn with identical distributions and correlation c.

)var(

),cov(

X

XXc ji

N

nnXS

1

)var()var( XNS

)var())1(()var( XcNNNS

ji

ji XXXNS ),cov()var()var(

What is the variance of S?

if c =0

otherwise

correlations can be neglected only if c << 1/N.

Asynchronous spikes vs.coincident spikes

2 input spikes in a noisy neuron

Vm

PSTH

Average extra number of spikes

2 coincident spikes

Coincidence sensitivity S = difference

A simple approach

w

pw

threshold

p coincident spikes

Inst

anta

neou

s cu

rren

tsex

pone

ntial

cur

rent

s

p spikes

S

Simulations

theory

Neurons are sensitive to coincidences in the balanced regime only

threshold

2 coincident spikes

w

2w

w

2w

threshold

2 coincident spikes

Balanced regime(= fluctuation-driven)

Oscillator regime(= mean-driven)

Quantitative results:2 spikes

Synaptic weight

Extr

a sp

ikes

Back

grou

nd n

oise

Quantitative results:p spikes

w

pw

threshold

p coincident spikes

Effect depends on p*w rather than p

Distributed synchrony

event event

Theory Simulation

Synchrony events involving p random synapses(w=0.5 mV)

Summary

• In the balanced regime, neurons are extremely sensitive to input correlations

• Correlations have negligible effects only if they are small compared to 1/N (N synapses)

• In fact, correlations undetectable in pair recordings can have tremendous postsynaptic effect

• We have a simple method to calculate the (stochastic) effect of coincidences

)var())1(()var( XcNNNS