architecture and equilibra 结构和平衡 chapter 6. 2002.12.4 2 chapter 6 architecture and...
TRANSCRIPT
Architecture and EquilibraArchitecture and Equilibra结构和平衡结构和平衡
Chapter 6
2002.12.4
2
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and EquilibriaPerface
lyaoynov stable theorem
I
I
S
L
I : 整个系统集合
S : 稳定系统集合
L : 可由李亚普诺夫函数判定稳定的系统集合
2002.12.4
3
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.1 6.1 Neutral Network As Stochastic Gradient syNeutral Network As Stochastic Gradient systemstem
Classify Neutral network model By their synaptic connection topolgies and by how learning modifies their connection topologies
pathwaysfeedbackorloopssynapticclosediffeedback
loopssynapticclosedNoifdfeedforwar
..2
..1
samplingstrainingunlabelleduselearningervisedUn
samplingstraining
oformationmembershipclassuselearningSupervised
:sup.2
inf:.1
synaptic connection topolgies
how learning modifies their connection topologies
2002.12.4
4
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.1 6.1 Neutral Network As Stochastic Gradient sNeutral Network As Stochastic Gradient systemystem
Gradi edescent
LMSBackPropagati on
Rei nforcement Leari ng
Recurrent BackPropagati on
Vetor Quanti zati on
Sel f -Organi zati on MapsCompeti tve l earni ngCounter-propagati on
RABAMBroeni an anneal i ng
ABAMART-2
BAM-Cohen-Grossberg ModelHopfi el d ci rcui t
Brai n-state- I n_BoxAdapti ve-Resonance
ART-1ART-2
Feedforward Feedback
Decode
Supervi sed
Unsupervi sed
Encode
Neural NetWork Taxonomy
2002.12.4
5
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.2 6.2 Global Equilibra:convergence and stabilitGlobal Equilibra:convergence and stabilityy
Neural network :synapses , neurons
three dynamical systems:
synapses dynamical systems
neuons dynamical systems
joint synapses-neurons dynamical systems
Historically,Neural engineers study the first or second neural network.They usually study learning in feedforward neural networks and neural stability in nonadaptive feedback neural networks. RABAM and ART network depend on joint equilibration of the synaptic and neuronal dynamical systems.
'
M'
X
),(''
MX
2002.12.4
6
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.2 6.2 Global Equilibra:convergence and stabilitGlobal Equilibra:convergence and stabilityy
Equilibrium is steady state (for fixed-point attractors)
Convergence is synaptic equilibrium.
Stability is neuronal equilibrium.
More generally neural signals reach steady state even though the activations still change.
We denote steady state in the neuronal field
Stability - Equilibrium dilemma :Neuron fluctuate faster than synapses fluctuate.
Convergence undermines stability
6.10M
6.20X
xF
36.0Fx
2002.12.4
7
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms
We shall prove that:
Competitve AVQ synaptic vector converge to pattern-class centroid. They vibrate about the centroid in a Browmian motion
jm
Competitve learning adpatively qunatizes the input pattern space
charcaterizes the continuous distributions of pattern.
nR
)(xp
centroidXX
AVQ^
2002.12.4
8
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgoritSynaptic convergence to centroids:AVQ Algorithmshms
7.6,
6.6....321
jiifjDiDKDDDDnR
nRPattern
The Random Indicator function
Supervised learning algorithms depend explicitly on the indicator functions.Unsupervised learning algorthms don’t require this pattern-class information.
Centriod
KDDDDIIII ,......,,
321
860
1)(
j
j
D Dxif
DxifxI
j
96)(
)(^
jD dxxpjD dxxxp
jx
Comptetive AVQ Stochastic Differential Equations
2002.12.4
9
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgoritSynaptic convergence to centroids:AVQ Algorithmshms
The Stochastic unsupervised competitive learning law:
106])[(
jjjjj nmxySm
We want to show that at equilibrium jjjj xmxm )E(or
116)( xISjDj
We assume
The equilibrium and convergence depend on approximation (6-11) ,so 6-10 reduces :
126])[(
jjDj nmxxIm j
2002.12.4
10
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms
Competitive AVQ Algorithms
miixmi ,,......1,)()0( 1. Initialize synaptic vectors:
2.For random sample ,find the closet(“winning”)synaptic
vector
)(tx
)(tm j
221
2.......
136)()(min)()(
m
iij
xxxwhere
txtmtxtm
3.Update the wining synaptic vectors by the UCL ,SCL,or DCL learning algorithm.
)(tm j
2002.12.4
11
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms
Unsupervised Competitive Learning (UCL)
156)()1(
146)]()([)()1(
jiiftmtm
tmtxctmtm
ii
jijj
}{ ic defines a slowly deceasing sequence of learning coefficient
)(samples10,000for000,10
11.0,instanceFor txt
ci
Supervised Competitive Learning (SCL)
176)]()([)(
)]()([)(
166)()())(()()1(
Djxiftmtxctm
Djxiftmtxctm
tmtxtxrctmtm
jij
jij
jjijj
2002.12.4
12
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgoritSynaptic convergence to centroids:AVQ Algorithmshms
Differential Competitive Learning (DCL)
196)()1(
186)]()())[(()()1(
jiiftmtm
tmtxtySctmtm
ii
jjjtjj
))1(( tyS jj denotes the time change of the jth neuron’s competitive
signal . In practice we only use the sign of (6-20)206))(())1(())1(( tyStyStyS jjjjjj
Stochastic Equilibrium and Convergence
Competitive synaptic vector coverge to decsion-class centrols.
May coverge to locally mixima.
2002.12.4
13
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithmSynaptic convergence to centroids:AVQ Algorithmss
AVQ centroid theorem:
if a competitive AVQ system converges,it converge to the centroid of the sampled decision class.
2161)^
(Pr mequilibriuatxmob jj
Proof. Suppose the jth neuron in Fy wins the actitve competition.
Suppose the jth synaptic vector codes for decision class jm jD
2260
jm
Suppose the synaptic vector has reached equilibrium
2002.12.4
14
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms
mean-zero is singalnoise
236
jj nmxImofbecause jDj
jj
j
D
Dj
Dj
D
Dj
jR
jD
j
mEx
concludestheoremcentroidAVQthe
xdxxp
dxxxpm
dxxpmdxxxp
dxxpmx
nEdxxpmxxI
mEo
nExpectatioTake
j
j
jj
j
n j
^
:
)(
)(
246)()(
)()(
)())((
:
^^
2002.12.4
15
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.3 6.3 Synaptic convergence to centroids:AVQ AlgorithSynaptic convergence to centroids:AVQ Algorithmsms
Arguments:
• The sptial and temporal integrals are approximate equal.
•The AVQ centriod theorem assumes that convergence occurs.
•The AVQ centroid convergence theorem ensure :
exponential convergence
2002.12.4
16
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem
AVQ Convergence Theorem:
Competitive synaptic vectors converge exponentially quikly to pattern-class centroids.
Proof.Consider the random quadratic form L
2562
1
0 0
2
n
i
m
j
iji )m(xL
The pattern vectors x do not change in time.
(still valid if the pattern vector x change in time)
2002.12.4
17
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem
296)(2))((
126)(
286)(
276
266
i jnmx
i jmxxI
nmxImofbecause
mmx
mm
L
mm
Lx
x
LL
ijijiijiD
ijijiDij
ij
i j
iji
ij
i j ij
ij
i j iji
i
i
j
j
The average E[L] as Lyapunov function for the sochastic competitice dynamical system.
Assume: Noise process is zero-mean and independence of the noise process with “signal”process ijm-x
2002.12.4
18
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem
316)()(
306][2
jDj
iji dxxpmx
LELE
gives
So ,on average by the learning law 6-12, 0)(
LE
If any synaptic vector move along its trajetory.
So, the competitive AVQ system is asymtotically stabel,and in gereral converges exponentially quickly to a locally equilibrium.
Suppose 0)(
LE If 0
jm Then every synaptic vector has
Reached equilibrium and is constant .
2002.12.4
19
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem
Since p(x) is a nonnegative weigth function.
The weighted integral of the learning difference
must equal zero :
iji mx
326)()( odxxpmxDj
ji
So equilibrium synaptic vector equal centroids.Q.E.D
2002.12.4
20
Chapter 6 Architecture and EquilibraChapter 6 Architecture and Equilibra 6.4 AVQ Convergence Theorem6.4 AVQ Convergence Theorem
Argument
• Total mean-squared error of vector quantization for the partition
• So the AVQ convergence theorem implies that
the class centroid, and asymptotically ,competitive synaptic vector-total mean-squared error.
kDD ,.....1
126])[(
jjDj nmxxIm jBy
The Synaptic vectors perform stochastic gradient desent on the mean-squared-error in pettern-plus-error space 1nR
In the sense :competitive learning reduces to stochostic gradient descent
2002.12.4
21
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks
Global stability is jointly neuronal-synaptics steady state.
Global stability theorems are powerful but limited.
Their power:
•their dimension independence
•nonlinear generality
•their exponentially fast convergence to fixed points.
Their limitation:
•do not tell us where the equilibria occur in the state space.
2002.12.4
22
Chapter 6 Architecture and EquilibraChapter 6 Architecture and Equilibra 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks
Stability-Convergence Dilemma
Stability-Convergence Dilemma arise from the asymmetry in neounal and synaptic fluctuation rates.
Neurons change faster than synapses change.
Neurons fluctuate at the millisecond level.
Synapses fluctuate at the second or even minute level.
The fast-changing neurons must balance the slow-changing synapses.
2002.12.4
23
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.5 6.5 Global stability of feedback neural networGlobal stability of feedback neural networksks
Stability-Convergence Dilemma
1.Asymmetry:Neurons in and fluctuate faster than the synapses in M.
2.stability: (pattern formation).
3.Learning:
4.Undoing:
the ABAM theorem offers a general solution to stability-convergence dilemma.
00
yx FandF
.000
MFandF yx
.000
yx FandFM
xF yF
2002.12.4
24
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem
The ABAM TheoremThe Hebbian ABAM and competitive ABAM models are globally stabel.
356
346)i()()(
336)()()(
1
1
jiijij
n
i
ijijjjjj
p
j
ijjjiiiii
SSmm
mxSybyay
mySxbxax
Hebbian ABAM model:
Competitive ABAM model , replacing 6-35 with 6-36
366
ijijij mSSm
2002.12.4
25
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem
If the positivity assumptions 0000 '' jiji SSaa
Then, the models are asymptotically stable,
and the squared activation and synaptic velocities decrease exponentially quickly to their equilibrium values:
0,0,0222
ijji myx
Proof. the proof uses the bounded lyapunov function L
3762
1)()()()( 2
0'
0'
i jijjj
j
yjjjii
i
xiii
i j
ijji mdbSdbSmSSL ji
2002.12.4
26
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem
386))((: dt
dx
dx
dFtxF
dt
dgivesationdifferentiofrulechainthe
i
ii
i j
jiij
i
ijij
j
jji j
ijjiii
i j
ijjiij
j i
ijijjji j
ijjiii
jijijjjjiii
ij
i j
ji
i
ijij
jjj
ijji
ii
SSm
mSbbSmSbaS
throughby
mSSm
mSbySmSbxS
mmybSxbS
mSSmSySmSxSL
iji
416)(
)()(
366346
406)(
)()(
396
2
2'2'
''
''
''
Make the difference to 6-37:
2002.12.4
27
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem
,0000 '' jiji SSaaofbecause .,0, estrajectorisystemalongLSo
To prove global stability for the competitve learning law 6-36
i jijjiijij
i
ijij
j
jji j
ijjiii
mSSmSS
mSbbSmSbaSL
426))((
)()( 2'2'
.0
1)(
00
))(()(
2
estrajectorialongL
SjifmS
Sjif
mSSmSSmSSm
iji
ijjiijijijjiij
We prove the stronger asymptotic stable of the ABAM models
with the positivity assumptions. 0000 '' jiji SSaa
2002.12.4
28
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.6 The ABAM Theorem6.6 The ABAM Theorem
4360
40622'2'
i jij
jj
j
j
ii
i
i myb
Sx
a
SL
Along trajectories for any nonzero change in any neuronal activation or any synapse.
Trajectories end in equilibrium points.
Indeed 6-43 implies:
4560
44600222
ijji
ijji
myxiff
myxiffL
The squared velocities decease exponentially quickly because of the strict negativity of (6-43) and ,to rule out pathologies .
Q.E.D
2002.12.4
29
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 structural stability of unsuppervised learning and RABA6.7 structural stability of unsuppervised learning and RABAMM
Is unsupervised learning structural stability?
Structural stability is insensivity to small perturbations
•Structural stability ignores many small perturbations.
•Such perturbations preserve qualitative properties.
Basins of attractions maintain their basic shape.
2002.12.4
30
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
Random Adaptive Bidirectional Associative Memories RABAM
Browian diffusions perturb RABAM model.
The differential equations in 6-33 through 6-35 now become stochastic differential equations, with random processes as solutions.
ijij
yj
xi
msynapsetheinprocessmotionbrowianB
FinneuroniththeinprocessmotionbrowianB
FinneuroniththeinprocessmotionbrowianB
:
:
.:
The diffusion signal hebbian law RABAM model:
2002.12.4
31
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
486()(
476)i()()(
466)()()(
)
1
1
ijjiijij
n
i
ijijjjjj
p
j
ijjjiiiii
dByjSxiSmdm
dbjdtmxSybyady
dBidtmySxbxadx
With the stochastic competitives law:
496])()[( ijijiijjij dBdtmxSySdm
2002.12.4
32
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
486()(
476)i()()(
466)()()(
)
1
1
ijjiijij
n
i
ijijjjjj
p
j
ijjjiiiii
dByjSxiSmdm
dbjdtmxSybyady
dBidtmySxbxadx
With the stochastic competitives law:
496])()[( ijijiijjij dBdtmxSySdm
2002.12.4
33
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria 6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
With noise (independent zero-mean Gaussian white-noise process).
the signal hebbian noise RABAM model:
546)(,)(,)(
5360)()()(
526)()(
516)i()()(
506)()()(
222
1
1
ijijjjii
ijji
ijjjiiijij
jn
i
ijijjjjj
i
p
j
ijjjiiiii
nVnVnV
nEnEnE
nySxSmm
nmxSybyay
nmySxbxax
2002.12.4
34
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
RABAM Theorem.
The RABAM model (6-46)-(6-48) or (6-50)-(6-54),
is global stable.if signal functions are strictly increasing and ampligication functions and are strictly postive, the RABAM model is asympotically stable.
ia jb
Proof. The ABAM lyapunov function L in (6-37) now defines
a random process. At each time t,L(t) is a random variable.
The expected ABAM lyapunov function E(L) is a lyapunov function for the RABAM.
556),,(....)( dxdydMMyxpLLELRABAM
2002.12.4
35
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
586]}[{}[{
]}[{][
576]}[{}[{
]}[{}
][][{
566
)()(
'
'
'
'2
2'2'
j'
i'
i j
jiijij
j i
ijijjj
i j
jjiii
i j
jiijij
j i
ijijjj
i j
jjiii
i j
jiij
j i
ijijjji j
ijjiii
i j
jiijij
j i
ijijji j
ijjii
SSmnEmSbSnE
miSbSnELE
SSmnEmSbSnE
miSbSnESSm
mSbaSmSbaSE
SSmm
mSbySmSbxSE
LELE
ABAM
2002.12.4
36
Chapter 6 Architecture and EquilibriaChapter 6 Architecture and Equilibria6.7 Structural stability of unsuppervised learning and RABA6.7 Structural stability of unsuppervised learning and RABAMM
Q.E.N
000)(0)(,
606)(
596]}{[)(}[{)(
]}[{)(][
model""
'
'
ABAMABAM
ABAM
ABAM
LorLasaccordingestrajectorialongLEorLESo
LE
noisemeanzero
SSmEnEmSbSEnE
miSbSEnELE
RABAMtheintermnoiseadditiveandsignaltheofindepence
i j
jiijij
j i
ijijjj
i j
jjiii
【 Reference 】[1] “Neural Networks and Fuzzy Systems -Chapter 6” P.221-26
1 Bart kosko University of Southern California.