8/6/2019 Framework SIMs ACC

1/6

Proceedings of the American Control ConferenceArlington, VA June 25-27, 2001

A F r a m e w o r k f o r S u b s p a c e I d e n t i f i c a t i o n M e t h o d s

R u i j ie S h i a n d J o h n F . M a c G r e g o rDep t . o f Chem ica l Eng inee r ing , McMas te r Un ive rs i ty, Hami l ton , ON L8S 4L7 , Canada

Emai l: sh i r@mcmas t e r. ca , macg reg@ mcm as te r. ca

A b s t r a c t

Simi lar i t ies and d i fferences among var ious subspaceiden t i f i c a t i on me thods (MOESP, N4SID and CVA) a r eexamined by put t ing them in a genera l regress ionframew ork. S ubspace ident i f ica t ion methods cons is t of threes teps : es t imat ing the predic table subspace for mul t ip lefuture steps, then extracting state variables from thissubspace and finally f i t t ing the est imated states to a statespace model . The major d i fferences among these subspaceident i f ica t ion methods l ie in the regress ion or pro jec t ion

meth ods u sed in the f i r s t step to remov e the effec t of thefu ture inputs on the fu ture outputs and thereby es t imate thepredic table subspace , and in the la tent var iable methodsused in the secon d s tep to ext rac t es t imates o f the s ta tes .

This paper compares the exis t ing methods and proposessome new va r i a t i ons by examin ing t hem in a commonframework involv ing l inear regress ion and la tent var iablees t imat ion . Limi ta t ions o f the var ious metho ds becom eapparent when examined in th is manner. S imula t ions areincluded to i l lustrate the ideas discussed.

1. I n t r o d u c t i o n

Subspace ident i f ica t ion methods (SIMs) have become qui tepopular in recent years . The key idea in SIMs is to es t imatethe s ta te var iables or the extended observabi l i ty matr ixdi rec t ly f rom the input an d output da ta . The most inf luent ia lmethods are CVA (Canonica l Var ia te Analys is , Lar imore ,1990) , MOESP (Mul t ivar iable Output Error S ta te space ,Verhaegen and Dew i lde , 1992 ) and N4SID (Numer i ca lSubspace Sta te-Space Sys tem IDent i f ica t ion , VanOverschee and De Moor, 1994) . These methods are sodi fferent in the i r a lgor i thms tha t i t i s hard to br ing themtogether an d ge t mo re ins ights on the essent ia l ideas and the

connec t ions among them. Howeve r, some e ffo r t ha s beenmade to cont ras t these methods .Viberg (1995) gave an overview of SIMs and c lass i f ied

them in to rea l iza t ion-based or d i rec t types , and a lso poin tedout the d i fferent ways to ge t sys tem matr ices v ia es t imateds ta tes or ex tended observabi l i ty matr ix . Van Overschee andDe Moor (1995) gave a un i fy ing t heo rem based on l owerorder approximat ion of an obl ique pro jec t ion . Here d i fferentme thods a r e v i ewed a s d i f f e r en t cho i ces o f row and co lum nweight ing matr ices for the reduced rank obl ique pro jec t ion .The bas ic s t ruc ture an d idea o f the i r theorem is based ont ry ing to cas t these methods in to the N4SID a lgor i thm. I t

focuses on the a lgor i thms ins tead of concepts and ideasbeh ind o f t he se m e thods .

In th is paper, SIMs are com pared by cas t ing them in to agenera l s ta t i s t ica l regress ion f ramework. The fundamenta ls imi lar i t ies and d i fferences among these SIMs is c lear lyshown in this stat ist ical framework. All the discussion inth is paper i s limi ted to the open loop case o f l inear t imeinvar iant (LTI) sys tem.

In next sec t ion , a genera l f ramework for SIMs wi l l bese t up f ir s t. T hen the fo l low ing two sec t ions wi l l d iscuss themajor par ts and how these methods f i t to the f ramework. A

simula t ion example fo l lows to i l lus t ra te the key poin ts . Thelas t sec t ion provides conclus ions and some appl ica t ionguidel ines .

2 . G e n e r a l S t a t i s t ic a l F r a m e w o r k f o r

S I M s

2 .1 Data Re la t ionsh ips in M ul t i -s tep S ta te-space Represen ta t ionA l inear de terminis t ic -s tochas t ic combined sys tem can berepresented in the fo l lowing s ta te space form:

xk+ 1 = A x ~ + B u ~ + w ~ (1 )Y k = C x ~ + D u k + N w ~ + v ~ (2 )

wh ere outpu ts Yk, inputs Uk and state variab les Xk are o fd imension l , m and n respect ive ly, and s tochas t ic var iableWk and Vk are o f proper d imensions and un-corre la ted wi theach o ther.

In order to ca tch the dynamics , SIMs use the mul t ip les teps of pas t da ta to re la te to the m ul t ip le s teps o f fu turedata. Fo r an arb i t ra ry t ime po in t k taken as the current t imepoint , al l the past p steps of the inpu t form s a vecto r Up, andthe current and the fu ture j l s teps of the input form s avector u_r. S imi lar sym bols for outpu t and noise var iables

( some a lgo ri t hms a s sume p - j ) :

Up = y p =

/ u k - = / |\ u~-i ) \ Yk-i

0 - 7 8 0 3 - 6 4 9 5 - 3 / 0 1 / $ 1 0 . 0 0 2 0 0 1 A A C C 3 6 7 8

8/6/2019 Framework SIMs ACC

2/6

I H k

Uk+l

U f = / U k +2 .

k.Uk+t'-I

Y--f

F or c onv e n i e nce , a l l t h e pos s ib l e_ _ U pfor d i ffe ren t k a recol lec ted in co lu mn s of Up wh ich i s the pas t input da ta se t ,sim ilar n ota tio ns fo r Yp, Uf, Yf, Wp, Vp, Wf an d Vf. A ll theposs ib le Xk for d i f fe ren t k a re co l lec ted in co lumn s of Xk.T h e r e l a t i o ns h ip s b e t ween t he se da t a s e t s and t h e s t a t eva r i ab l e s a r e a na l y z ed i n t he fo l l owing m u l t i -s t ep s t a t e -s pa ce r ep r e sen t a t i o n a s a gene ra l env i ronm en t f o r d i s cus s ingS I M s a n d t h e i r f r a m e w o r k .

B a sed o n equ a t i on (1 ) , ( 2 ) and t he above n o t a t i ons , t hef o l l ow i ng m u l t i - s t e p s t a t e - space mo de l f o r th e cu r r en ts ta tes , the pas t and fu ture output da ta can be obta ined (Xk_pis the in i t ia l s ta te sequ ence) :

X k = A P X k _ p+ ~ ' ~ p U p - Jr - f ~ s , p m p (3 )

Yp : F p X ~ _ p + H p U p + H , . p W p + V p (4 )

Y: : r ~ X ~ + H ~ U ~ + H , , j W j + V ~ (5 )Where ex tended cont ro l lab le mat r ices a re . (2p=[Ap-IB,Ap-2B ,. .. , A B , B ] , . ('~s ,p=[A - l , m p-2, . . . , A , / ] ,t h e ex t en de dobs erv abil i ty m atri x is F f= [C T, (CA ) T, (CA2) T, . . . , (CAf-1)T]Tand t he l o wer b loc k t r i angu l a r Toep l i t z ma t r i c e s Hf an d Hs , fare :

H f =

H , , /

D 0 0 -.- 0

CB D 0 . . . 0

oA B C B D . ..

" " " ' 0

D,CAi -2 B C A1-3 B CA r-4 B . . .

N 0 0 . . . 0

C N 0 . . . 0

oCA C N .. .

" " " " 0

NA/~-2 C Ar-3 CA! - 4 . . .

Ff and H f sh ow t h e e f f ec t o f cu r r en t s t a te s and t he f u tu r einputs on the fu ture outputs respec t ive ly

The resu l t o f subs t i tu t ing Xk_p f rom (4) to (3) i s (Fp+ ist he p seud o - inv e r se ) :

x~ : A~rp+ rp + (n~ - APrp+ Hp)U p + (n.,. p - APrp+H.,,p)W~ - A'rp+Vp

(6 )That i s , cur ren t s ta te se quen ce Xk ( therefore FrXk) i s a l inear

com binat io n o f the pas t da ta . FfXk i s the f ree evolu t ion ofc u r r en t ou t pu t s ( w i th no fu tu r e i npu t s ) and i n dep en den t o fthe sys tem matr ices . I t i s the par t o f fu ture output space in(5) tha t can be es t imated f rom the da ta re la t ionships .

S ys t em s t a t e s c a n be de f i ned a s " t he min im um amo un to f i n fo rma t i on abo u t t he pa s t h i s to ry o f a sy s t e m wh ich i srequi r ed to predic t the fu ture mo t ion " (Ast r /Dm, 1970) . Heret h e l i near c om b i na t i o n o f the t e rms on t he r i g h t h and s i de o f

eq ua t i o n ( 6 ) su m m a r i ze s t he ne ce s sa ry i n fo rma t ion i n t hepas t h i s tory to predic t the fu ture outputs of (5). Bysubs t i tu t ing (6) in to (5) , a l inear re la t ionship be tween thefu ture outputs and the pas t da ta as wel l as the fu ture inputsi s ob ta ined:

YS : r ' / . (f2p - AS'Fp+ Hp )U p + H.rU r + F1iA"F,,+Y,,

+ r / ( ~ , . , - A " F, , + H , , , , ) W p - F r A " F, , + V , + H , . j W s + ~+(7)

A l l t h e t e rms i n vo l v i ng t he p a s t f o r m the ba s i s f o r F rXkI t f U f i s t he e f f ec t o f the f u t u r e i npu t s and can be r e m o ved iHf i s k nown o r e s t ima t e d , a nd t h e fu tu r e no i s e t e r ms a r e

u np red i c t a b l e . On ly F fXk i s predic tab le f rom the pas t da tase t , and th is predic tab le subspace i s the fundamenta l basefor SIMs to es t imate s ta te sequence Xk or the observabi l i ty

matr ix Ff .Wi th a u to - c o r r e l a t ed i np u t s ,H f U f i s cor re la ted wi th the

pa s t d a t a an d t h e r e f o r e pa r t o f i t c an be ca l cu l a t ed f rom thepa s t da t a i f t h e i np u t au to - c o r r e l a t ion r em a ins un ch a nged .Ho w ev e r, i t i s n o t pa r t o f t he c aus a l it y e f f ec t t o be mode l edi n sy s t e m i de n t i f i c a t i on , a nd t h e r e f o r e sh ou ld no t be t akenin to accou nt for the predic t ion of Yf base d on the pa s t da taTh e i np u t au to - co r r e l a t i o n ma y g iv e d i f f i c u l t y i n e s t ima t i onof the predic tab le subsp ace and the s ta te var iab les .

2 .2 G e n e r a l S t a t is t ic a l F r a m e w o r k f o r S I M s

Each S IM loo ks qu i t e d i f f e r e n t f r om o the r s i n concep t ,c ompu ta t i on t o o l s a nd i n t e rp r e t a t i o n . The o r i g ina l MOESPdo es a Q R d e com p os i t i on on [U ; Yf] a nd t hen a SV D onpa r t o f t he R ma t r i x . Pa r t o f t he s i ng u l a r v ec to r ma t r i x i

t a ken a s F f , b a s ed on w h ic h A an d C m a t r i c e s a r e e s t ima t edand B a nd D a r e e s t im a t ed t h ro ugh a LS f i tt ing . N4S ID

projec ts Yf onto [Yp; Up; Uf] and does an S VD on the parcor responding to the pas t da ta , the r ight s ingular vec tors a res t im ated as s ta te var iab les and f i t to the s ta te space mo delN4 SI D i s in t e rp r e t ed i n t h e con ce p t o f non - s t a t i ona ryKa lman f i l t e r s . C VA use s C CA (Can on i ca l Co r r e l a t i onAn a l y s i s ) t o e s t i ma t e t he s t a t e va r i ab l e s ( c a l l ed memory )and f i t them to the s ta te space model . I t i s in te rpre ted inma x i m um l i ke l i h ood p r i nc ip l e . As fo r t he de t a i l eda lgo r i t hms ( r e f e r t o p ape r s ) , t h e d i f f e r ence be twee n t he seSIMs seems so la rge tha t i t i s hard to f ind the s imi la r i t iesb e t w e e n t h e m .

In f ac t, i f t h e b as i c i de a s b eh ind t h e se m e t hods a r e

sc ru t i n iz ed f rom the v i ewp o in t o f s t at is t ic a l r eg r e s s ion , an dth e c ompu ta t i on me th ods a r e an a l yz ed i n r eg r e s s ion t e rmsan d r e l a t ed t o e ach o the r, t he s e m e t hods a r e f ound t o beve ry s i m i l a r, an d fo l l ow t he s a m e f r amework . Thef r ame wo rk c ons i s t s o f t h r ee s te p s :

i ) Es t i m a t e t he p r e d i c t a b l e su bsp ace F fXk by al i nea r r eg r e s s i on me th od

i i ) Ext rac t s ta te var iab les f rom the es t imateds ubs pace by a l at en t va ri ab l e me tho d

3 6 7 9

8/6/2019 Framework SIMs ACC

3/6

i i i) Then fi t the estimated states to a state spacemodel .

The major differences among SIMs are in the first twosteps and the third step is the same. The original MOSEPalgorithm extracts Ff from the estimated subspace. HereMOESP is analyzed based on es t imated s ta tes that comefrom exactly the same subspace as Ff (also refer to VanOverschee and De Moor, 1995).

3 . E s t i m a t i o n o f t h e P r e d i c t a b l e

S u b s p a c e

3 .1 L inear Re gress ion fo r Hf to Es t imateFfX k

In SIM s, the pred ictable subspace FfXk should be firstestimated in order to have a basis for estimation of states Xkor F~ matrix. T he cen tral proble m is how to remo ve thefuture input effects H f U faw ay from Yf in (5) in order toobtain a better estimate of the predictable subspace F~Xk.The coeff ic ient matr ix Hf is unknown and needs to beestimated.

Hf show s the effects o f Uf on If , and consists of thefirst f steps of impulse w eights o n low er diagonals (SISO)or b lock weights on block lower d iagonals (MIMO). Thetrue Hf is a lower block triangular matrix. These features (orrequirements of Hf) are very informat ive; however, mostalgorithm s do not m ake full use of these features. Differenta lgor i thm uses d i fferent method to es t imate Hf f rom theinput and output data sets. There are quite a few ways forthis task; however, they all belong to the l inear regressionmethod.

Once H f is estimated, say / / / , Y f- ~ Uf is an estimation

of the predictable subspace. Th is estimation includes theeffects of the estimation errors in ~ and the effects of

future s tochast ic s ignals , which can be removed away byprojection to the past data. This projection procedure mayinduce some error; however, in most cases i t is less than theunpredictable future noise. Some subspace identificationmethods , such as N4SID, do the es t imat ion of Hf andprojection onto the past data sets in one step.

3 .2 M e t h o d s U s e d to E s t i m a t e H f

1. Regre ss ion Yfaga ins t U f ( M O E S P )

Since H f is the coef ficien t m atrix relating /.If to Yf, it isnatura l to t ry to get Hf by di rec tly performing LS regressionof Yf against Uf as in (5). A basic assum ption for anunbiased result is that the future inputs are un-correlatedwith the noise terms in (7), which will also include theeffect of state variables/ ' fXk in this case. This method givesan unbiased resul t only when the inputs are whi te noisesignals. Once Hf is estimated, the predictable subspace is

estimated as Yf-/2//Uf. The original MOESP uses this

method to es t imate Hf impl ic i t ly and the predic tablesubspace via QR decompo si t ion on [U ; Yf].

If the input sequences are auto-correlated, this meth odregresses part o f the state effect away and gives a biasedresult for the predictable subspace. SVD on this subspace

wil l g ives an asymptot ica l unbiased es t imat ion of ~;how ever, the estimation Of Xk will be biased.

2 . Reg ress ion Y f again s t [Yp; Up; Uf] (N4SID )

Based on (6) , we kno w FfXk in (5) can be es t imated by alinear comb ination of the past inputs Up and past ou tputs Yp.It is a natu ral choice to regre ss Yf again st [Yp; Up; Uf]. Here

the regression coefficient for Uf is an estimate of Hf (/2(r)

and the part corresponding to the past data is an estimationof the predictable subspace, w hich is equivalent to

projection Y~-/2/.1Uf onto the past data. This estimation will

have a sl ight bias if the input signals are auto-correlated.This bias will occur because o f the correlation betw een pastoutputs an d the past noise terms in (7).

This i s the method used in N4SID to es t imate Hf and

the predictable subspace. I t is realized by QR decompositionof [U ; Pio; Yf] (Pio=[Yp;U p ] ) .The PO-MOESP (1994, pas toutput (PO-) M OESP ) gives s imi lar results .

3 .Cons t ruc t i ng Hf f rom impu l se we igh ts o f AR X m ode l(CVA)The nature of Hf implies that i t can be con structed by thefirst f impulse bloc k weights. These im pulse weig ht blockscan be estimated from a simple model, such as an ARXmod el or FIR model , w hich can be ob ta ined by regress ing Ykagainst Uk (if De 0), past inputs (Up) and p ast outputs (Yp).

The predictable su bspace then is estimated as Yr ~ Uf.

It includes all the future noise. This is the method someCVA algor i thms use to es t imated Hf and the predic tablesubspace.

4 . Reg re s s ion ou t me thod

Uf can be regressed out of both sides of (7) by projectin g tothe orthogonal space of Uf, i .e. , by post multiplying bothside by Pufo=I-Ufr(ufufr)- lUf.This removes away the Ufterm from the equation, and the coefficient matrices for pastdata in (7) can be obtained by regressing YfPufo againstPioPufo. This result is equivalent to that from N4SID, andwas impl ied in Van Overschee and De Moor (1995) . Themethod has a lso been appl ied to the CVA method ( refer toVan Overschee and De Moor, 1995; Carette, 2000). Seenext section for more discussion.

An other similar approach is to regress past data p out ofboth sides of (7) (projecting to the orthog onal space o f Pio,post mul t ip l ied by P p o = I-P, oT (PIoPIoT)-'PIo)for theestimation of Hr. This turns ou t to be equ ivalent to theapproach of N4SID.

5 . In s t rumen ta l Va r i ab l e Me thod .

If there is a variable that is correlated to Uf but has n o

3 6 8 0

8/6/2019 Framework SIMs ACC

4/6

correlation with Xk and the future noise, an unbiased He canbe es t imated by the ins t rumenta l var iable ( IV) method basedon (5). For a uto-co rrelated inputs, Ue correlates with Xkthrough i ts correlation with Up, therefore the part of Ue,which has no correlation with past data, has no correlationwith Xk. This part o f Ue can be constructed b y regressing Upout of Ue, and take the residual as the IV. Once He ises t imated, the predic table subspace can be eas i ly es t imated.

Al l these es t imat ion m ethods are var iants of the l inearregression m ethod: they d iffer only in their choice of theindependent and d ependent var iables , of regress ionsequences , and the degree o f u t i l iza t ion of knowledge aboutthe fea tures of He. The key problem comes f rom thecorrelation be twe en Ue and Xk, whic h arises from auto-correlation of the input seque nce in the open loop case. Thees t imat ion accuracy (b ias and var iance) in each methoddepends on the input signal, the true model structure, andthe signal to noise ratio (SNR).

4 . E s t i m a t i o n o f S t a t e V a r i a b l e s

4 .1 L a t e n t Va r i a b l e M e t h o d s f o r S t a t e

E s t i m a t i o n

The predic table subspace es t imated by the l inear regress ionmethods of the las t sec t ion i s a h igh d imensional space(>>system order n) co nsist ing of highly correlated variables.If there we re no e stimatio n error, this subspace sh ould beonly of rank n and any n indep endent var iables in the dataset or their l inear combinations can be taken as statevar iables . However, the es t imat ion er ror genera l ly makesthe space full rank. Direct choice of any n variables willhave large es t imat ion er ror and lose the useful informat ion

in all other variables, which are highly correlated to the truestates. Extracting only n l inear combinations from thishighly corre la ted h igh-dimensional space and keeping asmuch informat ion as poss ib le wi l l be the most des i rable .This is exactly the general goal and the si tuation for whichla tent var iable methods were developed. Latent var iablemethods are therefore employed in a l l SIMs as themetho dology for es t imat ion of the s ta te variables f rom thepredic table subspace .

Latent var iables (LVs) are l inear combinat ions o f theoriginal (manifest) variables for optimization of a specificobjective. There are a variety of latent variable methods

based on different optimization objectives. In general terms,Pr incip le Component Analys is (PCA), Par t ia l Leas t Squares(PLS) , Canonical Corre la t ion Analys is (CCA) and ReducedRank Analys is (RRA) are la tent var iable methods tha tmaximize var iance , covar iance , corre la t ion and predic tablevar iance respect ive ly ( for de ta i l s refer to Burnham,e t a l . ,1996) . Different SIMs employ di fferent LVMs or use themin different ways to estimated state variables.

4 .2 M e t h o d s U s e d f o r S t a t e E s t i m a t i o n

1 . P C A ( M O S E P a n d N 4 S I D )

Both N4SID and MOESP ex t r ac tX k by doing PCA on thees t imated predic table subspace , which i s essent ia l ly a SVDprocedure . This impl ies assumpt ions tha t / - ' fX has a la rgervar ia t ion than tha t of the es t imat ion e r ror and the two par tsare uncorrelated. The first assum ption is well satisfied if thesignal to noise ration is large, and this ensures the first nPCs are the estimated state variables. The secondassum ption is essentially for the unbiasne ss o f theestimation, and this is not satisfied in case of auto-correlatedinputs.

The s ta te-based MO ESP ( in or ig inal a lgor ithm) d i rec t lyuses PCA (SVD) on the es t imated predic table subspace Yf-

Uf, whe re ~ is obtained by directly regressing Yf onto

Uf, and PCs are taken as estimated states. This estimatedpredictable subspace includes all the furore noise and is notpredic ted by the pas t da ta , and therefore PCA resul ts havelarge es t imat ion er rors and no guarantee of the

predic tabi l i ty. The PO-MOESP appl ied PCA to theprojec t ion of es t imated predic table subspace onto par t of thepast data space, therefore the result is generally improved.This method gives unbiased result for white noise inputs. Ifthe inputs are auto-correlated, the result will be biased.

N4SID appl ies PCA (SVD) on the par t of the projec t ionY f / [ Yp ' , U p ; U f ]corresponding to the pas t da ta . As ment ionedabove, th is resul t can be deem ed as the predic table subspace

projection Yc-/7/rUf on to the past data. Here @ is the third

block of the regressio n coefficien t matrix. In fact , i t can beshown that th is method is equivalent to performing RRA on

past data a nd Yr ~ Uf (for proof, see Shi, 2001). I t is clear

that the best predictabil i ty in N4SID is in the sense o f total

predictable variance of Yf-/2(.1.Ue based on the past data, and

this is assured by projection onto the past data, and at thesame t ime the fu ture noise i s removed. Therefore thees t imat ion er ror and bias of Xk f rom N4S ID are very smal lin general .

2 . C C A ( C VA )

CV A applies CC A on P~o=[Yp; Up] andY = Y f e = Y c [ I fU e ,

and the first n latent variables (CVs) from the past data setare estimates of Xk. By selecting the cano nical variates w ith

largest correlation as states, one is maximizing the relativevariation in Y in each dime nsion rather than the a bsolutevariation.

CCA can also be applied to the results result ing fromprojecting future inputs Ue out both the past data a nd thefuture outputs, i .e . PioPueo and YePueo; how eve r, the directresults of canonical variatesJ1P~oPueoare obviously b iasedestimation for Xk. Here the co efficient ma trix J~ should beapplied to the original past data to get state estimates:J~P~o.These estimates are no longer orthogonal. The result is

3 6 8 1

8/6/2019 Framework SIMs ACC

5/6

p r o ven t o g iv e t h e u nb i a sed s t a t e va r i ab l e e s t ima t e s ( f o rp r o o f , s e e Sh i , 2 001 ) . H ow eve r, s i nce pa r t o f th e s t a te s i gn a li s r em ov e d awa y b y r eg re s s ing Uf ou t wh i l e th e no i s e is

kept in tac t, the d a ta se t YfPuro has a wo rse SN R than Yf -~/ . I f

in genera l .

3 . O t h e r P o s s i b l e M e t h o d s

O the r LV M s , su c h a s PLS and RRA , a r e po s s i b l e cho ice sf o r s t a t e ex t r ac t i o n . Fo r examp le , RRA sho u ld p ro v id ee s t ima t e s o f t he s t a t e s ba sed on t he s ame ob j ec t iv e a sN 4SID , t ha t i s , s t a t e s wh ich max imiz ing t he va r i a nc eexp l a i ned i n t he p r e d i c t ab l e subspace . H owe v e r, i t s hou ldg iv e nu mer i ca l l y i m p roved e s t ima t e s s i nce i t d i r e c tl yob t a in s t he n LV s w h i ch exp l a in t he g r ea t e s t v a r i a nce i n thep r e d i c t ab l e sub spac e , r a t he r t han t he two- s t ep p ro ce du re o ft he f i r st pe r fo rm i ng an i l l - cond i t i oned l e a s t squa re s ( o b l i qu ep r o j ec t i on ) f o l l owe d b y PCA /SV D ( see Sh i, 2 001 ) .

S i nc e t h e o b j e c t i ve o f PLS i s t o mode l t h e va r i ance i nthe pas t da ta se t and the predic tab le subspace as wel l ast h e i r co r r e la t i o n , i t w i l l g ene ra l l y no t p rov ide m in ima l o r de rs ta te mo dels . In e ffec t , i t wi l l t ry to provid e s ta te vec tors fort h e j o in t i np u t / ou t p u t space ( r e f e r t o Sh i an d Ma cG rego r,2 000 ) .

Com bin a t i o ns o f the me tho ds u sed fo r the p r ed ic t a b l es u bspac e e s t i m a t i o n a nd t he m e thods u sed f o r th e s t a tev a r i ab l e e s t ima t i on c a n l e ad t o a who le s e t o f d i f f er en ts u bspac e i den t i f i c a t i o n me thods .

5 . S i m u l a t i o n E x a m p l eIn th is sec t ion , a s imula t ion example i s used to i l lus t ra te thed i s cus s e d p o in t s . T he example i s a s imp le1 st o rde r S ISOp r o ce s s w i th AR(1 ) no i s e , mode l ed a s :

0.2z -1 1Yk = ~ u k -~ ek

l_0 .8z -1 1-0 .9 5z -1T h e i n pu t s i gn a l i s a PRBS s igna l w i th sw i t ch ing t ime

p e r iod T~ =5 and m ag n i tude o f 4 .0 . 1000 d a ta po i n t s a reco l l e c t e d w i th va r ( ek ) = l . 0 , and SNR i s ab ou t 0 . 9 3 ( i nvar iance) . Bo th pas t an d fu ture lag s teps a re takes as 7 fore v e r y m e t h o d .

D i f f e r en t m e tho d s fo r e s t ima t i on o f the H f ma t r ix a r ea p p l i ed t o t h e s im u l a t i on example and compared t o t he t r u er e sul t . A rou gh co m par i son i s t o take t he me an o f e l emen t so n l ow e r d i ago n a l a s the e s t ima t ed impu l se w e i gh t . Theresul t s and to ta l absolu te e r rors a re l i s ted in Table 1 . Theresul t s by re gres s ing Yf d i rec t ly o nto Uf a re c lear ly thef a r t he s t f r om the t r ue va lue s becaus e o f t he b ia s r e s u l t ingf r om th e s t r o ng au t o - co r r e l a t i on i n t he i npu t P R BS s ign a l .O t he r m e t hod s g i ve r e s u l t s c l o se t o the t r ue va lue s .

Th e re a r e man y i ndexes fo r compa r i son o f t h ees t imated s ta tes to the t rue s ta tes . One quick index i s thec a non i c a l co r r e l a t i on c oe ff i c i en t s be tween e s t ima t e d s t a t e sand the t rue s ta tes ( see Table 2) . This g ives a c lear idea onh o w th e two spa c e s a r e cons i s t en t w i th e ach o the r. MOE S Pu s i ng t he d i r ec t r e g r e s s ionYe/Uf c lear ly g ives poor resu l t s .

S t a t e e s t ima t i on by PLS g i ve s r e l a t i ve ly deg raded r e su l t sc o m p a r e d w i t h C C A o r R R A b a s e d o n th e s a m e e s t i m a te dp r ed i c t ab l e sub spa ce b y AR X . The e s t ima t ed s t a t e s by o theme tho ds a r e m uch c lo se r t o t he t r ue s t a t e s . S imi l a rc onc lu s i ons a r e i nd i ca t e d b y t he sq ua red mu l t i p l ec o r r e l a t io n R 2, wh ic h sho ws h ow muc h t he t o t a l sumsqu a r e s o f th e t r ue s t a te s c a n b e exp l a i n ed by t h e e s t im a t eds ta tes ( two s ta tes a re sca led to uni t var iance) .

CC A resul t s base d on PioPufo andYfPufo and t hose ba sedo n P i o an d Yf-HfUf(us ing the t rue He) a re a l so compared .T he coe ff i c i e n t ma t r i x J 1 f ro m th e fo rmer i s d i f f e r en t f r omthat of the la tte r ( to la rge to show) . T he es t im ated s ta tesf r om the f o r m er me th od a r e n o t l i nea r comb ina t i ons o ft hose e s t i m a t ed by t he l a t t e r m e t hod , bu t t he y a r e ve ryclose.

I f the es t im ated s ta tes a re used for f i t t ing the s ta te spacemo de l , e ach m e t hod can be c omp a r ed by p lo t t i ng t he i re s t i m a t ed imp u l se r e spo nse s (F i g . l ) a nd t he i r e r ro r s (F ig .2 )MOES P g i ve s i s a p oo r r e su l t f o r t h i s e xamp le . The r e su lf r om S I M -AR X- P LS h as a l a rg e e r r o r bu t c an be impro vedt o m a t ch t h e o th e r s b y u s in g m ore LVs ( r e f e r t o Sh i an dMa cGr eg o r, 2 000 ) . T he r esu l t s f rom o the r S IM s a r e ve ryc l o se t o t he t r u e v a l ue s . T he so m ewh a t i r r egu l a r r e sponsef r o m t h e A R X m o d e l i s a l s o s h o w n f o r c o m p a r i s o n . A lS IMs g i ve smoo th r e spo nse by f i t t i ng t he LVs t o t he s t a t eequat ion .

6 . C o n c l u s i o n sAl tho u gh S I M s a r e qu it e d i f f e r en t in the i r concep t s anda lgo r i t h m s , t h ey fo l l o w th e s a me s t a t i s t i c a l f r amework s eup h e r e : ( 1 ) u s e o f a li ne a r r e g r e s s io n m e thod t o e s t ima t e / - /and the predic tab le subspace ; (2) use of a la ten t var iab le

me th od fo r e s t im a t i on o f a m in ima l s e t o f t he s t a tevar iab les ; and (3) then f i t t ing to the s ta te space model . Byd i s c us s in g t he S I M s i n t h is f r am ewo rk t he i r s im i l a ri t ie s anddi ffe rences can be c lear ly seen . I t a l so revea ls poss ib le newmeth o ds an d n ew com b in a t i on s o f ex i s t ing a pp r oaches fon ove l m e t hod s , su ch a s u s e o f the IV me th od fo r t hee s t im a t i on o f Hf , and u se o f o th e r l a ten t va r i ab le m e th odsRR A a nd PLS fo r s t a t e e s t ima t i o n .

ReferencesAstrOm, K .J.,Introduction to stochastic Control Theory,Academic

Press, 1970Burnham, A.J .R . Viveros and J.F. MacGregor,Frameworks for

Latent Va ria ble Multivariate Regression, Journal ofChem ometrics, V 10, pp.31-45, 1996

Carette, P., personal communication, notes for CVA, 2000Larimore, W. E., Canonical Va ria te Analysis for system

Identification, Filtering an d Adap tive Control,Proc. 29 hIEEE conference on Decision and Control, Vol.1, HonoluluHawaii, 1990

Larimore, W. E.,Optimal Reduced Rank Modeling, Prediction,Monitoring a nd Control Us ing Canonical Variate Analysis,preprints of AD CHE M, B anff, 1997

3 6 8 2

8/6/2019 Framework SIMs ACC

6/6

L j u n g , L . A n d M c K e l v e y, T.,Subspace identification fro m closedloop data, Signa l P rocess ing , V52 , 1996

Sh i , R . and J . MacGregor,Mo deling o f Dynamic Systems usingLatent Variable and Subspace M ethods,J . o f Chemomet r i c s ,V14 , pp .423-439 , 2000

Sh i , R ., Ph .D thes i s , McM as te r Un ive r s i ty, ON, C anada , 2001Van O verschee , P. and De Mo or, B . ,N4SID " Subspace Algorithms

for the Identification of Combined Deterministic-StochasticSystem,Au tom atica , Vol . 30, No. 1 , pp. 75-93, 1994

Van Overschee , P. and De Moor,B., A Unifying Theorem forThr ee Subspace Sy st em , Identif ication Algorithms,Au toma tica , Vol .31 . No. 12, pp. 1835-1864 , 1995

Verhaegen , M. and Dewi lde P. ,Subspace Model Identification,Part 1., The Output-error State-space model identificationclass of algorithms, In te rna t iona l Journa l o f Con t ro l , V56 ,pp .1187-1210 , 1992

Verhaegen , M. ,Identification of Deterministic Part of MIMO StateSpace Models Given in Innovations from Input-output Data,Automat i ca , V30 , pp .61-74 , 1994

Vibe rg , M. , Subspace-based Methods for the Identification ofLinear Time-invariant System,Automatica , V31, pp. 1835-1851, 1995

Tab le 1 the Impu l se We igh t s in Es t ima ted Hf

M e t h o d Tr u e

W l

w 2

w3

w 4

w5

w 6

w 7

Sum Abs . E r r.

0

0 .2

0.16

0 .128

0 .1024

0 .0819

0 .0655

0

Y f / U f Y ( [ P I o ; U f l A R X I V

0.1003 0.0159 0 0 .0191

0 .2716 0 .1951 0 .1997 0 .1953

0 .2203 0 .1585 0 .1546 0 .1617

0 .1770 0 .1035 0 .1086 0 .1137

0 .1626 0 .0728 0 .0863 0 .0811

0 .1613 0 .0668 0 .0602 0 .0750

0 .2135 0 .0510 0 .0610 0 .0559

0.5688 0.1061 0.067 5 0.0778

Tab le 2 Com par i son be tw een the Es t ima ted and the True S ta te s

M e t h o d M O E S P N 4 S I D C VA S I M - SI M - S I M -

Predictable Yf/Uf Yf/[PIo;Uf] A R XSubspaceE s t im a t e d P C A P C A C C AStates1s t CC 0 .8680 0 .99932 nd CC 0 .2599 0 .9623R 2 0 .4031 0 .9612

I V- A R X - A R X -C C A P L S R R A

I V A R X A R X

C C A P L S R R A

0.9997 0 .9995 0 .9500 0 .99930 .9613 0 .9600 0 .9122 0 .96180 .9605 0 .9590 0 .8667 0 .9606

I rr lo ul se ~ b y n: ~s f rcrn MOESP, N4S1Dand CVA0. 3 . . . . . . . . .

0.25

True0 .2 i . . .. .. . .. MOESP

............ N4SID/!~i~ O VA

o ~ i"~ t ', ,

0 5 il ',,),.,

o. ,:?.:,.,

, " ~ ' " ~ ' ~ , . . ~ . , , . . .

" - , . . . , . . . . . . . . . - . . . . . . . . . . .

' ' o ; . . . . . .5 1 1 20 25 30 35 40 45 50- ~ ( S a ~ i n g I rt e r~ t )

F i g. 1 Im p u l s e r e s p o n s e f r o m S I M s

M : d ~ i n g F ~ s d t s b y S I M sO.O35

i i i i J i i i i

~,.~ / ~.!:o'

0 I ! : ~ " : , , * / - "

~ i ~ i ' , , ~ ' ! l . : ir r G2 ; . . ! , "~ ....

~ -0.01 i\ . , , i! Trua

E

-0.015l-

IL l

-0.(32

. . . . O VA

... . . . . . S I ~

. .. .. .. .. .. . S IM -I V ~

: i

i , , i

I

5-0.(125 . . , , , ,

0 10 15 20 25 3o 35 4o 45

]irr~ (SaTl:lirg rto~i)

F i g . 2 E r r o r o n t h e i m p u l s e r e s p o n s e s

3 6 8 3