architecture and equilibria 结构和平衡

Architecture and Equilibria结构和平衡

Chapter 6

神经网络与模糊系统

学生：李琦导师：高新波

22003.11.19

6.1 Neutral Network As Stochastic Gradient system

Classify Neutral network model By their synaptic connection topologies and by how learning modifies their connection topologies

1. :

2. :

feedforward if no closed synaptic loops

feedback if closed synaptic loops or feedback pathways

1. :

2. :

Supervised learning use class membership information of

training samples

Unsupervised learning use unlabelled training samplings

synaptic connection topologies

how learning modifies their connection topologies

32003.11.19

6.1 Neutral Network As Stochastic Gradient system

Gradi ent descent

LMS BackPropagati on Rei nforcement Leari ng

Recurrent BackPropagati on

Vetor Quanti zati on

Sel f -Organi zati on Maps Competi tve l earni ng Counter-propagati on

RABAM Browni an anneal i ng Bol tzmann l earni ng ABAM ART-2 BAM-Cohen-Grossberg Model Hopfi el d ci rcui t Brai n-State- I n-A-Box Maski ng fi el dAdapti ve-Resonance ART-1 ART-2

Feedforward FeedbackDecode

Supervised

Unsupervised

Encode

Neural NetWork Taxonomy

42003.11.19

6.2 Global Equilibria: convergence and stability

Three dynamical systems in neural network:

synaptic dynamical system

neuronal dynamical system

joint neuronal-synaptic dynamical system

Historically,Neural engineers study the first or second neural network.They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.

M

x

( , )x M

52003.11.19

6.2 Global Equilibria: convergence and stability

Equilibrium is steady state (for fixed-point attractors)Convergence is synaptic equilibrium.Stability is neuronal equilibrium.

More generally neural signals reach steady state even though the activations still change. We denote steady state in the neuronal field :

Global stability:

Stability - Equilibrium dilemma :Neurons fluctuate faster than synapses fluctuate.Convergence undermines stability.

Μ = 0x = 0

xF xF 0

0,0 Mx

62003.11.19

6.3 Synaptic convergence to centroids: AVQ Algorithms

We shall prove that competitive AVQ synaptic vector converge exponentially quickly to pattern-class centroids and, more generally, at equilibrium they vibrate about the centroids in a Browmian motion.

jm

Competitive learning adaptively quantizes the input pattern space . Probability density function characterizes the continuous distributions of patterns in .

nR )(xpnR

72003.11.19


1 2 3....

,

knR D D D D

D D if i ji j

The Random Indicator function

Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorithms don’t require this pattern-class information.

Centriod of :

1( )

0j

j

Dj

if x DI x

if x D

( )

( )j

j

D

jD

xp x dxx

p x dx

Competitive AVQ Stochastic Differential Equations:

jD

82003.11.19


The Stochastic unsupervised competitive learning law:

( )[ ]j j j j jm S y x m n

We want to show that at equilibrium or E( )j j j jm x m x ( )

jj DS I xAs discussed in Chapter 4:

The linear stochastic competitive learning law: ( )[ ]

jj D j jI x x m nm The linear supervised competitive learning law:

( )[ ]

( ) ( )

( )

( )

j

j i

j j D j j

j D Di j

r I x x m n

r I x I x

m x

x

92003.11.19


The linear differential competitive learning law:

In practice:

[ ]j j j jm S x m n

sgn[ ][ ]

1 0

sgn[ ] 0 0

1 0

j j j jm y x m n

if z

z if z

if z

102003.11.19


Competitive AVQ Algorithms

miixmi ,,......1,)()0( 1. Initialize synaptic vectors:

2.For random sample , find the closest (“winning”) synaptic

vector :

)(tx

)(tm j ( ) ( ) min ( ) ( )j ii

m t x t m t x t

3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm.

)(tm j

2 2 21 ....... nwhere x x x gives the squared Euclidean norm of x

112003.11.19


Unsupervised Competitive Learning (UCL)

( 1) ( ) [ ( ) ( )]

( 1) ( )

jj j t

i i

m t m t c x t m t

m t m t if i j

{ }tc defines a slowly decreasing sequence of learning coefficient

For instance , 0.1 1 for 10,000 samples ( )10,000t

tc x t

Supervised Competitive Learning (SCL)

( 1) ( ) ( ( )) ( ) ( )

( ) [ ( ) ( )]

( ) [ ( ) ( )]

j j t j j

j t j j

j t j j

m t m t c r x t x t m t

m t c x t m t if x D

m t c x t m t if x D

122003.11.19


( )j jS y

]sgn[ jy

Differential Competitive Learning (DCL)

( 1) ( ) ( ( ))[ ( ) ( )]

( 1) ( )

j j t j j j

i i

m t m t c S y t x t m t

m t m t if i j

( ( ))j jS y t denotes the time change of the jth neuron’s competitive signal

( ( )) ( ( 1)) ( ( ))j j j j j jS y t S y t S y t

In practice we often use only the sign of the signal difference or , the sign of the activation difference.

132003.11.19

总迭代次数样本数

样本数决定计算时间及精度

总迭代次数可以人为设定？

终止迭代的条件是否不需要？

计算时间及精度可以人为设定？

T

0 1

( 1) ( ) [ ( ) ( )]

( ) ( 1)

t

j j t j

j j

kc c

T

m t m t c x t m t

m t m t

影响迭代步长

终止迭代条件：－

142003.11.19

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

T=10 T=20

T=30 T=40 T=100

基于 UCL 的 AVQ 算法

152003.11.19


Stochastic Equilibrium and Convergence

jm

)(xp

Competitive synaptic vector converge to decision-class centroids. The centroids may correspond to local maxima of the sampled but unknown probability density function

162003.11.19

AVQ centroid theorem:

If a competitive AVQ system converges, it converges to the centroid of the sampled decision class.

( ) 1j jProb m x at equilibrium

Proof. Suppose the jth neuron in wins the competition.

Suppose the jth synaptic vector codes for decision class jm jD

0jm

So

Suppose the synaptic vector has reached equilibrium:


YF

( ) 1 1 ( )( )j jD j j j DI x iff S by S I xy

172003.11.19

6.3 Synaptic convergence to centroids: AVQ Algorithms:

( )( ) ( )

( ) ( )

( ) ( )

( )

( )

( )[ ]n j j

j

j j

j

j

j

D j j j D j jR

jD

jD D

D

j j

D

Take Expectation

o E m

I x x m p x dx E n m

x m p x dx

xp x dx m p x dx

xp x dxm x

p x dx

I x x m n

In general the AVQ centroid theorem concludes that at equilibrium:

jj xmE ][

182003.11.19


Arguments:

•The AVQ centriod theorem applies to the stochastic SCL and DCL law.

• The spatial and temporal integrals are approximate equal.

•The AVQ centriod theorem assumes that stochastic convergence occurs.

192003.11.19

6.4 AVQ Convergence Theorem

AVQ Convergence Theorem:

Competitive synaptic vectors converge exponentially quickly to pattern-class centroids.

Proof.Consider the random quadratic form L:

2

0 0

1

2

n m

i iji j

L (x m )

Note:

The pattern vectors x do not change in time.

0L

202003.11.19


L equals a random variable at every time t. E[L] equals a deterministic number at every t. So we use the average E[L] as Lyapunov function for the stochastic competitive dynamical system.

2

( )

( )( ) ( )

( )[ ]j

j

i iji i ji ij

iji j ij

i ij ij j D j ji j

D i ij i ij iji j i j

L LL x m

x m

Lm

m

x m m m

I x x m x m n

I x x m n

212003.11.19


2[ ] ( ) ( ) 0j

i ijDj i

E L E L x m p x dx

So, the competitive AVQ system is asymptotically stable, and in general converges exponentially quickly to a locally equilibrium.Suppose .Then every synaptic vector has reached equilibrium and is constant (with probability one) if holds.

0jm

Assume: sufficient smoothness to interchange the time derivative and the probabilistic integral—to bring the time derivative “inside” the integral.

( ) 0E L

222003.11.19


Since p(x) is a nonnegative weight function, the weighted integral of the learning differences must equal zero : i ijx m

( ) ( ) 0j

jDx m p x dx

So, with probability one, equilibrium synaptic vector equal centroids. More generally, average equilibrium synaptic vector are centroids: jj xmE ][

232003.11.19


( ) ( ) 0j

jDx m p x dx ][LE

jm

Arguments:

The vector integral in equals the gradient of

with respect to .

So the AVQ convergence theorem implies that the class centroids-and, asymptotically ,competitive synaptic vector-minimize the mean-squared error of vector quantization.

242003.11.19

6.5 Global stability of feedback neural networks

Global stability is jointly neuronal-synaptic steady state.

Global stability theorems are powerful but limited.

Their power:

•their dimension independence

•nonlinear generality

•their exponentially fast convergence to fixed points.

Their limitation:

• not tell us where the equilibria occur in the state space.

252003.11.19


Stability-Convergence Dilemma

Stability-Convergence Dilemma arises from the asymmetry in neuronal and synaptic fluctuation rates.

Neurons change faster than synapses change.

Neurons fluctuate at the millisecond level.

Synapses fluctuate at the second or even minute level.

The fast-changing neurons must balance the slow-changing synapses.

262003.11.19


Stability-Convergence Dilemma

1.Asymmetry:Neurons in and fluctuate faster than the synapses in M.

2.stability: (pattern formation).

3.Learning:

4.Undoing:

The ABAM theorem offers a general solution to stability-convergence dilemma.

0 0X YF and F

0 0 0.X YF and F M

0 0 0.X YM F and F

XF YF

272003.11.19

6.6 The ABAM Theorem

ij ij i j i jm m S S S S

1

1

( ) ( ) ( )

( ) ( ) ( )

( ) ( )

p

i i i i i j j ijj

n

j j j j j i i iji

ij ij i i j j

x a x b x S y m

y a y b y S x m

m m S x S y

Hebbian ABAM model:

Competitive ABAM model (CABAM):( ) ( )ij j j i i ijm S y S x m

Differential Hebbian ABAM model:

Differential competitive ABAM model:

ij j i ijm S S m

282003.11.19


The ABAM Theorem: The Hebbian ABAM and competitive ABAM models are globally stable.

We define the dynamical systems as above.

If the positivity assumptions hold, then the models are asymptotically stable, and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:

0,0,0,0 jiji SSaa

0,0,0 222 ijji myx

292003.11.19


Proof. The proof uses the bounded lyapunov function L:

: ( ( )) ii

i

dxd dFthe chain rule of differentiation gives F x t

dt dx dt

' ' 2

0 0

1( ) ( ) ( ) ( )

2

i jx yi ji j ij i i i i j j j j ij

i j i j i j

L S S m S b d S b d m

' '

' '

' '

' 2 ' 2 2

( ) ( ) ( )

( ) ( ) ( )

i j i

i i j ij j j i ij i j iji j j i i j

i j ij iji i j jj

i i i j ij j j j i ij ij i j iji j j i i j

i i i j ij j j j i ij ij i ji j j i j

L S x S m S y S m S S m

S b x S b y m m

S x b S m S y b S m m S S m

S a b S m S a b S m m S S

0, 0, 0, 0, 0, .

i

i j i jbecause of a a S S So L along system trajectories

This proves global stability for signal Hebbian ABAMs.

302003.11.19


ij j i ijm S S m for the competitive learning law:

' 2 ' 2( ) ( ) ( )( )i i i j ij j j j i ij j i ij i j iji j j i i j

L S a b S m S a b S m S S m S S m

2

0 0( ) ( )( )

( ) 1

0 .

j

ij i j ij j i ij i j ij

i ij j

if Sm S S m S S m S S m

S m if S

L along trajectories

We assume that behaves approximately as a zero-one threshold.

jS

This proves global stability for the competitive ABAM system.

312003.11.19

' '

''2 2 2

( ) ( ) ( )

0


jii j ij

i j i ji j

L S x b S m S y b S m m S S m

SSx y m

a b

2 2 20 0

0

i j ij

i j ij

L iff x y m

iff x y m

along trajectories for any nonzero change in any neuronal activation or any synapse.

This proves asymptotic global stability.


Also for signal Hebbian learning:

(Higher-Order ABAMs, Adaptive Resonance ABAMs, Differential Hebbian ABAMs)

322003.11.19

6.7 structural stability of unsupervised learning and RABAM

•Structural stability is insensitivity to small perturbations.

•Structural stability allows us to perturb globally stable feedback systems without changing their qualitative equilibrium behavior.

•Structural stability differs from the global stability, or convergence to fixed points.

•Structural stability ignores many small perturbations. Such perturbations preserve qualitative properties.

332003.11.19


The signal Hebbian diffusion RABAM model:

Random Adaptive Bidirectional Associative Memories RABAM

ijji BandBB ,,

XF YF ijm

Brownian diffusions perturb RABAM models.

Suppose denote Brownian-motion (independent Gaussian increment) processes that perturb state changes in the ith neuron in ,the jth neuron in ,and the synapse ,respectively.

( ) ( ) ( )

( ) ( ) ( )

( ) ( )

i i i i i j j ij ij

j j j j j i i ij ji

ij ij i i j j ij

dx a x b x S y m dt dB

dy a y b y S x m dt dB

dm m dt S x S y dt dB

342003.11.19


( )[ ( ) ]ij j j i i ij ijdm S y S x m dt dB

With the stochastic competitives law:

(Differential Hebbian, differential competitive diffusion laws)

The signal-Hebbian noise RABAM model:

2 2 2

( ) ( ) ( )

( ) ( ) ( )

( ) ( )

0

, ,

i i i i i j j ij ij

j j j j j i i ij ji

ij ij i i j j ij

i j ij

i i j ij

x a x b x S y m n

y a y b y S x m n

m m S x S y n

E n E n E n

V n

352003.11.19


The RABAM theorem ensures stochastic stability.

In effect, RABAM equilibria are ABAM equilibria that randomly vibrate. The noise variances control the range of vibration. Average RABAM behavior equals ABAM behavior.

RABAM Theorem.

The RABAM model above is global stable. If signal functions are strictly increasing and amplification functions and are strictly positive, the RABAM model is asymptotically stable.

ia jb

362003.11.19


Proof. The ABAM lyapunov function L :

defines a random process. At each time t, L(t) is a random variable.

The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM system.

( ) .... ( , , )RABAML E L L p x y M dxdydM

' ' 2

0 0

1( ) ( ) ( ) ( )

2

i jx yi ji j ij i i i i j j j j ij

i j i j i j

L S S m S b d S b d m

372003.11.19


' '

' 2 ' 2 2

' '

( ) ( ) ( )

( ) ( ) ( )

( ) ( )


i i i j ij j j j i ij ij i ji j j i i j

i i i j ij j j j i ij ii j j i

E L E L

E S x b S m S y b S m m S S m

E S a b S m S a b S m m S S

E n S b S m E n S b S m E n

'

'

( )

( )

( ) ( )

0 0 0 0

j ij i ji j

ABAM i i i j iji j

j j j i ij ij ij i jj i i j

ABAM

ABAM ABAM

m S S

E L E n E S b S m

E n E S b S m E n E m S S

E L

So E L or E L along trajectories according as L or L

382003.11.19


ix

ix

Noise-Saturation Dilemma:How neurons can have an effective infinite dynamical range whenthey operate between upper and lower bounds and yet not treat small input signals as noise: If the are sensitive to large inputs,then why do not small inputs get lost in internal system noise? If the are sensitive to small inputs, then why do they not all saturateat their maximum values in response to large inputs?

392003.11.19


RABAM Noise Suppression Theorem:

As the above RABAM dynamical systems converge exponentially quickly, the mean-squared velocities of neuronal activations and synapses decrease to their lower bounds exponentially quickly:

Guarantee: no noise processes can destabilize a RABAM if the noise processes have finite instantaneous variances (and zero mean).

(Unbiasedness Corollary, RABAM Annealing Theorem)

2 2 2 2 2 2, ,i i j j ij ijE x E y E m

402003.11.19

Thank you!

architecture and equilibria 结构和平衡

Documents