ramdom signals

8/9/2019 Ramdom Signals

1/23

C H A P T E R 9

Random

Processes

INTRODUCTION

MuchofyourbackgroundinsignalsandsystemsisassumedtohavefocusedontheeffectofLTI systemsondeterministic signals, developing tools foranalyzing thisclass of signals and systems, and using what you learned in order to understand

applications

in

communication

(e.g.,

AM

and

FM

modulation),

control

(e.g.,

sta-bilityoffeedbacksystems),andsignalprocessing(e.g.,filtering). ItisimportanttodevelopacomparableunderstandingandassociatedtoolsfortreatingtheeffectofLTI systemson signalsmodeledastheoutcomeofprobabilisticexperiments, i.e.,aclassofsignalsreferredtoasrandomsignals(alternativelyreferredtoasrandomprocesses or stochastic processes). Such signals play a central role in signal andsystem design and analysis, and throughout the remainder of this text. In thischapterwedefinerandomprocessesviatheassociatedensembleofsignals,andbe-gintoexploretheirproperties. Insuccessivechaptersweuserandomprocessesasmodels for randomoruncertain signals thatarise in communication, controlandsignalprocessingapplications.

9.1

DEFINITION

AND

EXAMPLES

OF

A

RANDOM

PROCESS

InSection7.3wedefinedarandomvariableXasafunctionthatmapseachoutcomeofaprobabilisticexperimenttoarealnumber. Inasimilarmanner,arealvaluedCT or DT random process, X(t) or X[n] respectively, is a function that mapseachoutcomeofaprobabilisticexperimenttoarealCTorDTsignalrespectively,termed the realization of the random process in that experiment. For any fixedtime instant t = t0

or n = n0, the quantities X(t0) and X[n0] arejust randomvariables. Thecollectionofsignalsthatcanbeproducedbytherandomprocessisreferredtoastheensembleofsignals intherandomprocess.

EXAMPLE9.1 RandomOscillators

Asanexampleofarandomprocess, imagineawarehousecontainingN harmonicoscillators, eachproducinga sinusoidalwaveformof some specificamplitude, fre-quency,andphase,allofwhichmaybedifferent forthedifferentoscillators. Theprobabilisticexperimentthatresultsintheensembleofsignalsconsistsofselectingan oscillator according to some probability mass function (PMF) that assigns aprobabilitytoeachofthenumbersfrom1toN,sothattheithoscillatorispicked

c 161Alan

V.

Oppenheim

and

George

C.

Verghese,

2010


2/23

162

Chapter

9

Random

Processes

Amplitude

X(t;)

t0 t

FIGURE

9.1 Arandomprocess.

withprobabilitypi. Associatedwitheachoutcomeofthisexperiment isaspecificsinusoidalwaveform.

In Example 9.1, before an oscillator is chosen, there is uncertainty about whatthe amplitude, frequency and phase of the outcome of the experiment will be.Consequently,forthisexample,wemightexpresstherandomprocessas

X(t) =Asin(t+)

where the amplitude A, frequency and phase are all random variables. Thevalue X(t1)at some specific time t1 isalsoa randomvariable. In thecontextofthis experiment, knowing the PMF associated with each of the numbers 1 to Ninvolved inchoosinganoscillator,aswellasthespecificamplitude,frequencyandphaseofeachoscillator,wecoulddeterminetheprobabilitydistributionsofanyof

the

underlying

random

variables

A,

,

or

X(t1)

mentioned

above.

Throughoutthisandlaterchapters,wewillbeconsideringmanyotherexamplesofrandomprocesses. What is importantatthispoint,however, istodevelopagoodmentalpictureofwhatarandomprocessis. Arandomprocessisnotjustonesignalbutratheranensembleofsignals,as illustratedschematically inFigure9.2below,forwhichtheoutcomeoftheprobabilisticexperimentcouldbeanyofthefourwave-forms indicated. Eachwaveform isdeterministic, but the process isprobabilisticor randombecause it isnotknown a prioriwhichwaveformwillbegeneratedbytheprobabilisticexperiment. Consequently,priortoobtainingtheoutcomeoftheprobabilisticexperiment,manyaspectsofthesignalareunpredictable,sincethereisuncertaintyassociatedwithwhichsignalwillbeproduced. Aftertheexperiment,oraposteriori,theoutcomeistotallydetermined.

If

we

focus

on

the

values

that

a

random

process

X(t)

can

take

at

a

particularinstantoftime,sayt1 i.e.,ifwelookdowntheentireensembleatafixedtime

whatwehave isarandomvariable,namelyX(t1). Ifwe focusontheensembleofvaluestakenatanarbitrarycollectionoffixedtimeinstantst1 < t2


3/23

Section

9.1

Definition

and

examples

of

a

random

process

163

X(t) = x (t)

t t1 2

FIGURE

9.2 RealizationsoftherandomprocessX(t)

canbe thoughtof asa familyofjointlydistributed randomvariables indexedbyt (or n in theDT case). A fullprobabilistic characterization of this collectionofrandomvariableswould require thejointPDFsofmultiple samplesof the signal,takenatarbitrarytimes:

a

X(t) = x (t)b

X(t) = x (t)c

X(t) = x (t)d

t

t

t

t

fX(t1),X(t2), ,X(t)(x1, x2, , x)

forallandallt1, t2, , t.

Animportantsetofquestionsthatarisesasweworkwithrandomprocessesinlater

chapters

of

this

book

is

whether,

by

observing

just

part

of

the

outcome

of

a

randomprocess,wecandeterminethecompleteoutcome. Theanswerwilldependonthe

detailsofthe randomprocess,but ingeneral theanswer isno. For somerandomprocesses, having observed the outcome in a given time interval might providesufficientinformationtoknowexactlywhichensemblememberwasdetermined. Inothercasesitwouldnotbesufficient. Wewillbeexploringsomeoftheseaspectsinmoredetail later,butweconcludethissectionwithtwoadditionalexamplesthat

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


4/23

164

Chapter

9

Random

Processes

furtheremphasizethesepoints.

EXAMPLE9.2 Ensembleofbatteries

ConsideracollectionofNbatteries,eachprovidingonevoltageoutofagivenfinitesetofvoltagevalues. Thehistogramofvoltages(i.e.,thenumberofbatterieswithagivenvoltage) isgiven inFigure9.3. Theprobabilisticexperiment is tochoose

Number of

Batteries

Voltage

FIGURE

9.3 HistogramofbatterydistributionforExample9.2.

oneofthebatteries,withtheprobabilityofpickinganyspecificonebeingN

1 , i.e.,theyareallequallylikelytobepicked. Alittlereflectionshouldconvinceyouthat

if

we

multiply

the

histogram

in

Figure

9.3

by

N

1

,

this

normalized

histogram

will

represent(orapproximate)thePMFforthebatteryvoltageattheoutcomeoftheexperiment. Since thebattery voltage isa constant signal, this corresponds toarandomprocess, and in fact is similar to theoscillator examplediscussed earlier,butwith=0and=0,sothatonlytheamplitudeisrandom.

For this exampleobservationof X(t)atanyone time is sufficient information todeterminetheoutcomeforalltime.

EXAMPLE

9.3

Ensemble

of

coin

tossers

Consider

N

people,

each

independently

having

written

down

a

long

random

stringofonesandzeros,witheachentrychosenindependentlyofanyotherentryintheir

string(similartoasequenceofindependentcointosses). Therandomprocessnowcomprises this ensemble of strings. A realization of the process is obtained byrandomlyselectingaperson(andthereforeoneoftheN stringsofonesandzeros),following which the specific ensemble member of the random process is totallydetermined. Therandomprocessdescribed inthisexample isoftenreferredtoas

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


5/23

Section

9.1

Definition

and

examples

of

a

random

process

165

theBernoulliprocessbecauseof theway inwhich the stringofonesand zeros isgenerated(byindependentcoinflips).

Now suppose thatperson showsyouonly the tenth entry in the string. Canyoudetermine (orpredict)theeleventhentry fromjustthat information? Becauseofthemanner inwhichthestringwasgenerated,theanswerclearly isno. Similarlyif the entire past history up to the tenth entry was revealed to you, could youdeterminetheremainingsequencebeyondthetenth? Forthisexample,theanswerisagainclearlyno.

While the entire sequence hasbeendeterminedby thenatureof the experiment,partialobservationofagivenensemblemember isingeneralnotsufficienttofullyspecifythatmember.

Ratherthanlookingatthenthentryofasingleensemblemember,wecanconsiderthe randomvariable corresponding to thevalues from the entire ensembleat the

nth

entry.

Looking

down

the

ensemble

at

n

=

10,

for

example,

we

would

would

seeonesandzeroswithequalprobability.

In theabovediscussionwe indicated and emphasized that a random process canbe thoughtofasa familyofjointlydistributed randomvariables indexedby torn. Obviouslyitwouldingeneralbeextremelydifficultorimpossibletorepresentarandomprocessthisway. Fortunately,themostwidelyusedrandomprocessmodelshavespecialstructurethatpermitscomputationofsuchastatisticalspecification.Also,particularlywhenweareprocessingoursignalswithlinearsystems,weoftendesigntheprocessingoranalyzetheresultsbyconsideringonlythefirstandsecondmomentsoftheprocess,namelythefollowingfunctions:

Mean:

X(ti) =

E[X(ti)],

(9.1)

Auto-correlation: RXX(ti, tj) =E[X(ti)X(tj)], and (9.2)

Auto-covariance: CXX(ti, tj) =E[(X(ti)X(ti))(X(tj)X(tj))]

=RXX(ti, tj)X(ti)X(tj). (9.3)

Thewordauto(which issometimewrittenwithoutthehyphen,andsometimesdroppedaltogether to simplify the terminology)here refers to the fact thatbothsamplesinthecorrelationfunctionorthecovariancefunctioncomefromthesameprocess;weshallshortlyencounteranextensionofthisidea,wherethesamplesaretakenfromtwodifferentprocesses.

One case in which the first and second moments actually suffice to completelyspecify the process is in the case of what is called a Gaussian process, definedas

a

process

whose

samples

are

always

jointly

Gaussian

(the

generalization

of

the

bivariateGaussiantomanyvariables).

Wecanalsoconsidermultiplerandomprocesses,e.g.,twoprocesses,X(t)andY(t).Forafullstochasticcharacterization ofthis,weneedthePDFsofallpossiblecom-binationsofsamplesfromX(t), Y(t). WesaythatX(t)andY(t)areindependentifeverysetofsamplesfromX(t)isindependentofeverysetofsamplesfromY(t),

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


6/23

166

Chapter

9

Random

Processes

sothatthejointPDFfactorsasfollows:

fX(t1), ,X(tk),Y(t), ,Y(t)(x1,

, xk, y1,

, y) 1

=fX(t1), ,X(tk

)(x1, , xk).fY(t), ,Y(t)(y1, , y). (9.4)1

Ifonlyfirstandsecondmomentsareof interest,theninadditiontotheindividualfirstand secondmomentsof X(t)and Y(t) respectively, weneed to consider thecrossmomentfunctions:

Cross-correlation:

RXY(ti, tj) =E[X(ti)Y(tj)], and (9.5)

Cross-covariance:CXY(ti, tj) =E[(X(ti)X(ti))(Y(tj)Y(tj))]

=RXY(ti, tj)X(ti)Y(tj). (9.6)

IfCXY(t1, t2)=0forallt1, t2,wesaythattheprocessesX(t)andY(t)areuncor-

related.

Note

again

that

the

term

uncorrelated

in

its

common

usage

means

thattheprocesseshavezerocovarianceratherthanzerocorrelation.

Note that everything we have said above can be carried over to the case of DTrandomprocesses,exceptthatnowthesampling instantsarerestrictedtobedis-crete time instants. In accordance with our convention of using square brackets[ ] around the time argument for DT signals, we will write X[n] for the meanof a random process X[ ] at time n; similarly, we will write RXX[ni, nj] for thecorrelationfunction involvingsamplesattimesni

andnj

;andsoon.

9.2 STRICT-SENSESTATIONARITY

Ingeneral,wewouldexpectthatthejointPDFsassociatedwiththerandomvari-

ables

obtained

by

sampling

a

random

process

at

an

arbitrary

number

k

of

arbitrarytimeswillbetimedependent, i.e.,thejointPDF

fX(t1), ,X(tk

)(x1, , xk)

willdependonthespecificvaluesoft1, , tk. IfallthejointPDFsstaythesame underarbitrarytimeshifts,i.e.,if

fX(t1

), ,X(tk)(x1, , xk) =fX(t1+), ,X(tk+)(x1, , xk) (9.7)

forarbitrary,thentherandomprocessissaidtobestrictsensestationary(SSS).Saidanotherway, fora strictsense stationaryprocess, the statisticsdependonlyontherelativetimesatwhichthesamplesaretaken,notontheabsolutetimes.

EXAMPLE9.4 Representingan i.i.d. process

ConsideraDTrandomprocesswhosevaluesX[n]mayberegardedasindependentlychosenateachtimenfromafixedPDFfX

(x),sothevaluesareindependentandidenticallydistributed,therebyyieldingwhat iscalledan i.i.d. process. Suchpro-cesses are widely used in modeling and simulation. For instance, if a particular

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


7/23

Section

9.3

Wide-Sense

Stationarity

167

DT communication channel corrupts a transmitted signal with added noise thattakes independentvaluesateach time instant,butwithcharacteristics that seem

unchanging

over

the

time

window

of

interest,

then

the

noise

may

be

well

modeled

asani.i.d. process. Itisalsoeasytogenerateani.i.d. processinasimulationenvi-ronment,providedonecanarrangearandomnumbergeneratortoproducesamplesfromaspecifiedPDF(andthereareseveralgoodwaystodothis). Processeswithmorecomplicateddependenceacrosstimesamplescanthenbeobtainedbyfilteringorotheroperationsonthei.i.d. process,asweshallseeinthenextchapter.

Forsuchani.i.d. process,wecanwritethejointPDFquitesimply:

fX[n1],X[n2], ,X[n](x1, x2, , x) =fX(x1)fX(x2) fX(x) (9.8)

foranychoiceofandn1, , n. TheprocessisclearlySSS.

9.3 WIDE-SENSE STATIONARITY

Ofparticularuse tous isa less restricted typeof stationarity. Specifically, if themeanvalue X

(ti) is independentof timeand theautocorrelation RXX

(ti, tj

) orequivalentlytheautocovarianceCXX

(ti, tj

)isdependentonlyonthetimedifference(titj), then the process is said to be widesense stationary (WSS). Clearly aprocess that is SSS is also WSS. For a WSS randomprocess X(t), therefore, wehave

X(t) =X (9.9)

RXX

(t1, t2) =RXX

(t1

+, t2

+)forevery

=

RXX (t1

t2,

0)

.

(9.10)

(NotethatforaGaussianprocess(i.e.,aprocesswhosesamplesarealwaysjointlyGaussian)WSS impliesSSS,becausejointlyGaussianvariablesareentirelydeter-minedbythetheirjointfirstandsecondmoments.)

Two random processes X(t) and Y(t) arejointly WSS if their first and secondmoments (including the crosscovariance) are stationary. In this case we use thenotationRXY()todenoteE[X(t+)Y(t)].

EXAMPLE9.5 RandomOscillatorsRevisited

ConsideragaintheharmonicoscillatorsasintroducedinExample9.1, i.e.

X(t;A,)=Acos(0t+)

where A and are independent random variables, and now 0

is fixed at someknownvalue.

If isactuallyfixedat theconstantvalue 0, theneveryoutcome isof the formx(t) =Acos(0t+0),anditisstraightforward toseethatthisprocessisnotWSS

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


8/23

168

Chapter

9

Random

Processes

(andhencenotSSS).Forinstance,ifAhasanonzeromeanvalue,A =0,thentheexpectedvalueoftheprocess,namelyAcos(0t+0), istimevarying. Toargue

that

the

process

is

not

WSS

even

when

A =

0,

we

can

examine

the

autocorrelation

function. Notethatx(t)isfixedatthevalue0forallvaluesoftsuchthat0t+0

isanoddmultipleof/2,andtakesthevaluesAhalfwaybetweensuchpoints;thecorrelationbetweensuchsamplestaken/0

apartintimecancorrespondinglybe0(intheformercase)orE[A2](inthe latter). Theprocess isthusnotWSS.

Ontheotherhand,ifisdistributeduniformly in [, ],then 1

X(t) =A cos(0t+)d= 0, (9.11) 2

CXX (t1, t2) =RXX(t1, t2)

=E[A2]E[cos(0t1

+)cos(0t2

+)]

E[A2

]

=

cos(0(t2

t1))

,

(9.12)2

sotheprocess isWSS. Itcanalsobeshown tobeSSS,though this isnot totallystraightforwardtoshowformally.

To simplify notation for a WSS process, we write the correlation function asRXX (t1t2); the argument t1t2 is referred to as the lag at which the corre-lation is computed. For the most part, the random processes that we treat willbeWSSprocesses. When consideringjustfirstand secondmomentsandnot en-tire PDFs orCDFs, itwillbe less important todistinguish between the randomprocessX(t)andaspecificrealizationx(t)of itsoweshallgoonestepfurtherin simplifyingnotation, byusing lower case letters todenote the randomprocessitself.

We

shall

thus

talk

of

the

random

process

x(t),

and

in

the

case

of

a

WSS

processdenote itsmeanby x and itscorrelation function E{x(t+)x(t)}byRxx(). Correspondingly, forDTwellrefertotherandomprocessx[n]and(intheWSScase)denoteitsmeanbyx anditscorrelationfunctionE{x[n+m]x[n]}byRxx[m].

9.3.1 SomePropertiesofWSSCorrelationandCovarianceFunctions

It iseasilyshownthatforrealvaluedWSSprocessesx(t)andy(t)thecorrelationandcovariancefunctionshavethefollowingsymmetryproperties:

Rxx() =Rxx(), Cxx() =Cxx() (9.13)

Rxy() =Ryx(), Cxy

() =Cyx(). (9.14)

Weseefrom(9.13)thattheautocorrelationandautocovariancehaveevensymme-try. SimilarpropertiesholdforDTWSSprocesses.

Another important property of correlation and covariance functions follows fromnotingthatthecorrelationcoefficientoftworandomvariableshasmagnitudenot

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


9/23

Section

9.4

Summary

of

Definitions

and

Notation

169

exceeding 1. Applying this fact to the samples x(t) and x(t+) of the randomprocessx( )directly leadstotheconclusionthat

Cxx(0) Cxx() Cxx(0). (9.15)

Inotherwords, theautocovariance functionneverexceeds inmagnitude itsvalueattheorigin. Addingx

2 toeachtermabove,wefindthefollowinginequalityholds

forcorrelationfunctions:

Rxx(0)+2x2

Rxx() Rxx(0). (9.16)

In Chapter 10 we will demonstrate that correlation and covariance functions arecharacterized by the property that their Fourier transforms are real and non-negative at all frequencies, because these transforms describe the frequency dis-tributionoftheexpectedpower intherandomprocess. Theabovesymmetrycon-straintsandboundswill then followasnaturalconsequences,but theyareworthhighlightingherealready.

9.4 SUMMARYOFDEFINITIONSAND NOTATION

Inthissectionwesummarizesomeofthedefinitionsandnotationwehavepreviouslyintroduced. As in Section 9.3, we shall use lower case letters to denote random

processes,

since

we

will

only

be

dealing

with

expectations

and

not

densities.

Thus,

with x(t)and y(t)denoting (real) randomprocesses,we summarize the followingdefinitions:

mean : (t)

(9.17)x =E{x(t)}

autocorrelation : (t1, t2)

(9.18)Rxx =E{x(t1)x(t2)}

crosscorrelation : (t1, t2)

(9.19)Rxy =E{x(t1)y(t2)}

autocovariance : (t1, t2)

(t1)][x(t2)x(t2)]}Cxx =E{[x(t1)x

=Rxx(t1, t2)x(t1)x(t2) (9.20)

crosscovariance : (t1, t2)

(t1)][y(t2)y(t2)]}Cxy =E{[x(t1)x

=Rxy

(t1, t2)x(t1)y

(t2) (9.21)

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


10/23

170

Chapter

9

Random

Processes

strict-sense

stationary

(SSS): alljointstatisticsforx(t1),x(t2),...,x(t)forall>0andallchoicesofsampling instantst1,,t

depend

only

on

the

relative

locations

of

sampling

instants.

wide-sense

stationary

(WSS): x(t)isconstantatsomevaluex,andRxx(t1,t2) isafuncti

jointlywide-sensestationary:

of(t1t2)only,denoted inthiscasesimplybyRxx(t1t2);henceCxx(t1,t2) isafunctionof(t1t2)only,andwrittenasCxx(t1t2).x(t)andy(t)areindividuallyWSSandRxy(t1,t2) isafunctionof(t1t2)only,denotedsimplybyRxy(t1

t2);henceCxy(t1,t2)isafunctionof(t1

t2)only,andwrittenasCxy(t1

t2).

ForWSSprocesseswehave, incontinuoustimeandwithsimplernotation,

Rxx(

) =

E{x(t

+

)x(t)}

=

E{x(t)x(t

)}

(9.22)

Rxy() =E{x(t+)y(t)}=E{x(t)y(t)}, (9.23)

andindiscretetime,

Rxx[m] =E{x[n+m]x[n]}=E{x[n]x[nm]} (9.24)

Rxy [m] =E{x[n+m]y[n]}=E{x[n]y[nm]}. (9.25)

Weusecorresponding(centered)definitionsandnotationforcovariances:

Cxx(), Cxy(), Cxx[m], andCxy[m].

ItisworthnotingthatanalternativeconventionusedelsewhereistodefineRxy()

asRxy =E{x(t)y(t+)}.()

Inournotation,thisexpectationwouldbedenotedbyRxy(). Itsimportanttobecarefultotakeaccountofwhatnotationalconventionis being followed when you read this material elsewhere, and you should also beclearaboutwhatnotationalconventionweareusing inthistext.

9.5 FURTHEREXAMPLES

EXAMPLE9.6 Bernoulliprocess

The

Bernoulli

process,

a

specific

example

of

which

was

discussed

previously

inExample9.3,isanexampleofani.i.d. DTprocesswith

P(x[n]=1)=p (9.26)

P(x[n] =1)=(1p) (9.27)

andwith thevalue at each time instant n independent of the valuesatallother

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


11/23

Section

9.5

Further

Examples

171

timeinstants. Asimplecalculationresultsin

E

{x[n]}

= 2p

1 =

x (9.28)

1 m= 0

E{x[n+m]x[n]}=

(2p1)2 m= 0(9.29)

Cxx[m] =E{(x[n+m]x)(x[n]x)} (9.30)

={1(2p1)2}[m] = 4p(1p)[m]. (9.31)

EXAMPLE9.7 Randomtelegraphwave

A

useful

example

of

a

CT

random

process

that

well

make

occasional

reference

to is the random telegraph wave. A representative sample function of a randomtelegraphwaveprocessisshowninFigure9.4. Therandomtelegraphwavecanbedefinedthroughthefollowingtwoproperties:

t

x(t)

+1

1

FIGURE

9.4

One

realization

of

a

random

telegraph

wave.

1. X(0)=1withprobability0.5.

2. X(t)changespolarityatPoissontimes,i.e.,theprobabilityofksignchangesinatimeintervaloflengthT is

(T)keTP(ksignchangesinanintervaloflengthT) = . (9.32)

k!

Property 2 implies that the probability of a nonnegative, even number of signchanges inanintervaloflengthT is

(T

)k

1 + (1)k

(T

)k

P(a

nonnegative

even

#

of

sign

changes)

=

eT =

eTk! 2 k!

k=0

k=0

k even

(9.33)Usingthe identity

(T)kT

e =

k!k=0

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


12/23

172

Chapter

9

Random

Processes

equation(9.33)becomes

P(a

nonnegative

even

#

of

sign

changes)

=

eT (eT

+

eT

)

21

= (1+e2T). (9.34)2

Similarly,theprobabilityofanoddnumberofsignchangesinanintervaloflengthT

is 1(1e2T). Itfollowsthat2

P(X(t)=1)=P(X(t) = 1X(0)=1)P(X(0)=1)|

+P(X(t) = 1|X(0)=1)P(X(0)=1)

1= P(even#ofsignchangesin[0, t])

21

+

P(odd

#

of

sign

changes

in

[0, t])

2

1

1

1

1

1(1e2t)= (1+e2t) + = . (9.35)

2 2 2 2 2

NotethatbecauseofPropertyI,theexpressioninthelastlineofEqn. (9.35)isnotneeded,sincethelinebeforethatalreadyallowsustoconcludethattheansweris 12

:sincethenumberofsignchanges inany intervalmustbeeitherevenorodd,theirprobabilitiesaddupto1,soP(X(t)=1)= 12. However,ifProperty1isrelaxedtoallowP(X(0)=1)=p0

= 21,thentheabovecomputationmustbecarriedthrough

tothe last line,andyieldstheresult

(1e2t)P(X(t)=1)=p0 (1+e2t) +(1p0) =

1

1

1

1+(2p01)e

2t

.2 2 2

(9.36)

ReturningtothecasewhereProperty1holds,soP(X(t)=1),weget

X

(t) = 0, and (9.37)

RXX

(t1, t2) =E[X(t1)X(t2)]

= 1P(X(t1) =X(t2))+(1)P(X(t1) = X(t2))

=e2|t2t1| . (9.38)

Inotherwords,theprocess isexponentiallycorrelatedandWSS.

9.6

ERGODICITY

Theconceptofergodicity issophisticatedandsubtle,buttheessential idea isde-scribedhere. Wetypicallyobservetheoutcomeofarandomprocess(e.g.,werecordanoisewaveform)andwanttocharacterizethestatisticsoftherandomprocessbymeasurementsononeensemblemember. Forinstance,wecouldconsiderthetimeaverageofthewaveformtorepresentthemeanvalueoftheprocess(assumingthis

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


13/23

Section

9.7

Linear

Estimation

of

Random

Processes

173

mean isconstant foralltime). Wecouldalsoconstructhistogramsthatrepresentthefractionoftime(ratherthantheprobabilityweightedfractionoftheensemble)

that

the

waveform

lies

in

different

amplitude

bins,

and

this

could

be

taken

to

reflect

the probability density across the ensemble of the value obtained at a particularsamplingtime. Iftherandomprocessissuchthatthebehaviorofalmosteverypar-ticular realizationover time is representativeof thebehaviordown theensemble,thentheprocess iscalledergodic.

Asimpleexampleofaprocessthat isnotergodic isExample9.2,anensembleofbatteries. Clearly,forthisexample,thebehaviorofanyrealizationisnotrepresen-tativeofthebehaviordowntheensemble.

Narrowernotionsofergodicitymaybedefined. Forexample, ifthetimeaverage

1

T

x=T

2T Tx(t)dt (9.39)lim

almost always (i.e. foralmost every realization oroutcome) equals the ensembleaverage X

, then the process is termed ergodic in the mean. It can be shown,for instance, that a WSSprocess with finite variance at each instant and with acovariance function thatapproaches0 for large lags isergodic in themean. Notethat a (nonstationary) process with timevarying mean cannot be ergodic in themean.

Inourdiscussionof randomprocesses, wewillprimarilybe concernedwithfirstand secondorder moments of random processes. While it is extremely difficultto determine in general whether a random process is ergodic, there are criteria(specified in terms of the moments of the process) that will establish ergodicityin the mean and in the autocorrelation. Frequently, however, such ergodicity is

simply

assumed

for

convenience,

in

the

absence

of

evidence

that

the

assumptionis not reasonable. Under this assumption, the mean and autocorrelation can be

obtainedfromtimeaveragingonasingleensemblemember,throughthefollowingequalities:

1T

E{x(t)}= lim x(t)dt (9.40)T

2TT

and

1T

E{x(t)x(t+)}= lim x(t)x(t+)dt (9.41)T2T

T

A

random

process

for

which

(9.40)

and

(9.41)

are

true

is

referred

as

secondorder

ergodic.

9.7 LINEARESTIMATIONOFRANDOM PROCESSES

Acommonclassofproblemsinavarietyofaspectsofcommunication,controlandsignalprocessing involvestheestimationofonerandomprocessfromobservations

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


14/23

174

Chapter

9

Random

Processes

of another, or estimating (predicting) future values from the observation of pastvalues. Forexample,itiscommonincommunicationsystemsthatthesignalatthe

receiver

is

a

corrupted

(e.g.,

noisy)

version

of

the

transmitted

signal,

and

we

would

like to estimate the transmitted signal from the received signal. Other exampleslie in predicting weather and financial data from past observations. We will betreatingthisgeneraltopicinmuchmoredetailinlaterchapters,butafirstlookatitherecanbebeneficialinunderstandingrandomprocesses.

Weshallfirstconsiderasimpleexampleof linearpredictionofarandomprocess,thenamoreelaborateexampleoflinearFIRfilteringofanoisecorruptedprocesstoestimatetheunderlyingrandomsignal. Weconcludethesectionwithsomefurtherdiscussionof thebasicproblemof linear estimationofone randomvariable frommeasurementsofanother.

9.7.1 LinearPrediction

Asasimple illustrationof linearprediction,consideradiscretetimeprocessx[n].Knowing thevalueat time n0

we may wish to predictwhat the value will be msamples intothefuture,i.e. attimen0+m. We limitthepredictionstrategytoalinearone, i.e.,with x[n0+m]denotingthepredictedvalue,werestrictx[n0+m]tobeoftheform

x[n0+m] =ax[n0] +b (9.42)

and choose the prediction parameters a and b to minimize the expected valueofthesquareoftheerror, i.e.,chooseaandbtominimize

=E{(x[n0+m]x[n0+m])2} (9.43)

or

=E{(x[n0+m]ax[n0]b)2}. (9.44)

Tominimizewesettozero itspartialderivativewithrespecttoeachofthetwoparametersandsolvefortheparametervalues. Theresultingequationsare

E{(x[n0+m]ax[n0]b)x[n0]}=E{(x[n0+m]x[n0+m])x[n0]}= 0(9.45a)

E{x[n0

+m]ax[n0]b}=E{x[n0

+m]x[n0

+m]}= 0.(9.45b)

Equation (9.45a) states that theerror x[n0

+m]x

[n0

+m]associatedwith theoptimalestimateisorthogonaltotheavailabledatax[n0]. Equation(9.45b)states

that

the

estimate

is

unbiased.

Carryingoutthemultiplicationsandexpectationsintheprecedingequationsresultsinthefollowingequations,whichcanbesolvedforthedesiredconstants.

Rxx[n0+m, n0]aRxx[n0, n0]bx[n0]=0 (9.46a)

x[n0+m]ax[n0]b= 0. (9.46b)

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


15/23

Section

9.7

Linear

Estimation

of

Random

Processes

175

IfweassumethattheprocessisWSSsothatRxx[n0+m, n0] =Rxx[m],Rxx[n0, n0] =Rxx[0],andalsoassumethatitiszeromean,(x =0),thenequations(9.46)reduce

to

a=Rxx[m]/Rxx[0] (9.47)

b=0 (9.48)

sothatRxx[m]

x[n0+m] =Rxx[0]

x[n0]. (9.49)

Iftheprocess isnotzeromean,then itiseasytoseethat

Cxx[m]x

[n0+m] =x+

Cxx[0](x[n0]x). (9.50)

Anextensionofthisproblemwouldconsiderhowtodopredictionwhenmeasure-mentsofseveralpastvaluesareavailable. Ratherthanpursuethiscase,weillustratenextwhattodowithseveralmeasurementsinaslightlydifferentsetting.

9.7.2 LinearFIRFiltering

Asanotherexample,whichwewilltreatinmoregeneralityinchapter11onWienerfiltering, consideradiscretetime signal s[n] thathasbeen corruptedbyadditivenoised[n]. Forexample,s[n]mightbeasignaltransmittedoverachannelandd[n]thenoiseintroducedbythechannel. Thereceivedsignalr[n]isthen

r[n] =s[n] +d[n]. (9.51)

Assume that both s[n] and d[n] are zeromean random processes and are uncor-related. At the receiver we would like to process r[n] with a causal FIR (finiteimpulseresponse)filtertoestimatethetransmittedsignals[n].

d[n]

s[n] s[n]

r[n] h[n]

FIGURE

9.5

Estimating

the

noise

corrupted

signal.

Ifh[n]isacausalFIRfilteroflengthL,then

L1

s[n] =h[k]r[nk]. (9.52)k=0

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


16/23

176

Chapter

9

Random

Processes

Wewouldliketodeterminethefiltercoefficientsh[k]tominimizethemeansquareerrorbetweens[n]ands[n], i.e.,minimizegivenby

=E(s[n]s[n])2L1

=E(s[n]

h[k]r[nk])2. (9.53)k=0

To determine h, we seth[m]

= 0 for each of the L values of m. Taking this

derivative,weget

=E{2(s[n]

h[k]r[nk])r[nm]}

h[m]k

=E{2(s[n]s

[n])r[nm]}

= 0

m

= 0,

1,

, L

1

(9.54)

whichistheorthogonalityconditionweshouldbeexpecting: theerror(s[n]s[n])associatedwiththeoptimalestimateisorthogonaltotheavailabledata,r[nm].

Carrying out the multiplications in the above equations and taking expectationsresultsin

L1h[k]Rrr [mk] =Rsr[m], m= 0,1, , L1 (9.55)

k=0

Eqns. (9.55)constituteLequationsthatcanbesolved fortheLparameters h[k].With r[n] = s[n] +d[n], it is straightforward to show that Rsr[m] = Rss[m] +Rsd[m]andsinceweassumedthats[n]andd[n]areuncorrelated,thenRsd[m]=0.Similarly,Rrr [m] =Rss[m] +Rdd[m].

These

results

are

also

easily

modified

for

the

case

where

the

processes

no

longer

havezeromean.

9.8 THEEFFECTOFLTI SYSTEMS ONWSSPROCESSES

Yourpriorbackground insignalsandsystems,and intheearlierchaptersofthesenotes,hascharacterizedhowLTIsystemsaffecttheinputfordeterministicsignals.

Wewill see in laterchaptershow the correlationpropertiesofa randomprocess,andtheeffectsofLTIsystemsontheseproperties,playanimportantroleinunder-standinganddesigningsystems forsuchtasksasfiltering,signaldetection, signalestimation and system identification. We focus in this section on understandinginthetimedomainhowLTIsystemsshapethecorrelationpropertiesofarandomprocess.

In

Chapter

10

we

develop

a

parallel

picture

in

the

frequency

domain,

af-terestablishingthatthefrequencydistributionoftheexpectedpowerinarandomsignal isdescribedbytheFouriertransformoftheautocorrelationfunction.

ConsideranLTIsystemwhoseinputisasamplefunctionofaWSSrandomprocessx(t),i.e.,asignalchosenbyaprobabilisticexperimentfromtheensemblethatcon-stitutestherandomprocessx(t);moresimply,wesaythattheinputistherandom

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


17/23

Section

9.8

The

Effect

of

LTI

Systems

on

WSS

Processes

177

process x(t). TheWSS input ischaracterizedby itsmeanand itsautocovarianceor(equivalently)autocorrelationfunction.

Amongotherconsiderations,weareinterestedinknowingwhentheoutputprocessy(t)i.e.,theensembleofsignalsobtainedasresponsestothesignalsintheinputensemblewillitselfbeWSS,andwanttodetermineitsmeanandautocovarianceorautocorrelationfunctions,aswellasitscrosscorrelationwiththeinputprocess.ForanLTIsystemwhose impulseresponse ish(t),theoutputy(t) isgivenbytheconvolution

+ +y(t) = h(v)x(tv)dv= x(v)h(tv)dv (9.56)

foranyspecificinputx(t)forwhichtheconvolutioniswelldefined. Theconvolutioniswelldefinedif,forinstance,theinputx(t)isboundedandthesystemisboundedinputboundedoutput(BIBO)stable, i.e. h(t) isabsolutely integrable. Figure9.6indicates

what

the

two

components

of

the

integrand

in

the

convolution

integral

may

looklike.

x(v)

v

h(t - v)

t v

FIGURE

9.6 Illustrationofthetwoterms inthe integrandofEqn. (9.56)

Ratherthanrequiringthateverysamplefunctionofourinputprocessbebounded,it will suffice for our convolution computations below to assume that E[x2(t)] =Rxx(0)isfinite. Withthisassumption,andalsoassumingthatthesystemisBIBOstable, weensure that y(t) isawelldefined randomprocess,and that the formalmanipulationswe carryoutbelow for instance, interchangingexpectationandconvolution can all bejustified more rigorously by methods that are beyondour scopehere. In fact, theresultsweobtaincanalsobeapplied,whenproperlyinterpreted, to cases where the input process does not have a bounded secondmoment,

e.g.,

when

x(t)

is

socalled

CT

white

noise,

for

which

Rxx() =(). TheresultscanalsobeappliedtoasystemthatisnotBIBOstable,aslongasithasawelldefinedfrequencyresponseH(j),asinthecaseofanideallowpassfilter,forexample.

We can use the convolution relationship (9.56) to deduce the first and secondorderpropertiesofy(t). Whatweshallestablishisthaty(t)isitselfWSS,andthat

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


18/23

178

Chapter

9

Random

Processes

x(t) and y(t) are in factjointly WSS. We will also develop relationships for theautocorrelationoftheoutputandthecrosscorrelationbetweeninputandoutput.

First, consider themeanvalueof theoutput. Taking the expectedvalueofbothsidesof(9.56),wefind +

E[y(t)]=E h(v)x(tv)dv += h(v)E[x(tv)]dv

+= h(v)x

dv

+

=x

h(v)dv

=H(j0)x

=y

. (9.57)

Inotherwords,themeanoftheoutputprocessisconstant,andequalsthemeanofthe inputscaledbythetheDCgainofthesystem. This isalsowhattheresponseofthesystemwouldbeifits inputwereheldconstantatthevaluex.

Theprecedingresultandthelinearityofthesystemalsoallowustoconcludethatapplyingthezero-meanWSSprocessx(t)x totheinputofthestableLTIsystemwould result in the zero-meanprocess y(t)y at theoutput. This fact willbeuseful below in converting results that are derived for correlation functions intoresultsthatholdforcovariancefunctions.

Nextconsiderthecrosscorrelationbetweenoutputand input:

+

E{y(t+)x(t)}=E h(v)x(t+v)dv x(t) + = h(v)E{x(t+v)x(t)}dv. (9.58)

Sincex(t)isWSS,E{x(t+v)x(t)}=Rxx(v),so +E{y(t+)x(t)}= h(v)Rxx(v)dv

=h()Rxx()

=Ryx(). (9.59)

Note that the crosscorrelation depends only on the lag between the samplinginstantsof theoutputand inputprocesses, notonboth and theabsolute timelocationt. Also,thiscrosscorrelationbetweentheoutputandinputisdeterminis-ticallyrelatedtotheautocorrelationofthe input,andcanbeviewedasthesignalthatwouldresultifthesysteminputweretheautocorrelationfunction,asindicatedinFigure9.7.

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


19/23

Section

9.8

The

Effect

of

LTI

Systems

on

WSS

Processes

179

Ryx()Rxx()

h()

FIGURE

9.7 RepresentationofEqn. (9.59)

Wecanalsoconcludethat

Rxy() =Ryx() =Rxx()h() =Rxx()h(), (9.60)

wherethesecondequalityfollowsfromEqn. (9.59)andthefactthattimereversingthetwo functions inaconvolution results intimereversalofthe result,whilethelastequalityfollowsfromthesymmetryEqn. (9.13)oftheautocorrelationfunction.

Theaboverelationscanalsobeexpressed intermsofcovariance functions,ratherthanintermsofcorrelationfunctions. Forthis,simplyconsiderthecasewheretheinput to the system is thezeromeanWSSprocess x(t)x,withcorrespondingzeromeanoutputy(t)y . Sincethecorrelationfunctionforx(t)x isthesameasthecovariancefunctionforx(t),i.e.,since

Rxx,xx() =Cxx(), (9.61)

the results above hold unchanged when every correlation function is replaced bythecorrespondingcovariancefunction. Wethereforehave,for instance,that

Cyx() =

h(

)

Cxx()

(9.62)

Nextweconsidertheautocorrelationoftheoutputy(t): +

E{y(t+)y(t)}=E h(v)x(t+v)dv y(t)

+= h(v)E{x(t+v)y(t)}dv

Rxy(v)

+

= h(v)Rxy(v)dv

=h()Rxy()

=Ryy(). (9.63)

Note that theautocorrelationof theoutputdependsonlyon , andnotonboth and t. Puttingthistogetherwith theearlierresults,weconclude that x(t)and

y(t)arejointlyWSS,asclaimed.

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


20/23

180

Chapter

9

Random

Processes

Thecorrespondingresultforcovariancesis

Cyy() =h()Cxy(). (9.64)

Combining(9.63)with(9.60),wefindthat

Ryy() =Rxx() h()h() =Rxx()Rhh(). (9.65)

h()h()=Rhh()

ThefunctionRhh()istypicallyreferredtoasthedeterministicautocorrelationfunction

ofh(t),and isgivenby +Rhh() =h()h() = h(t+)h(t)dt. (9.66)

For

the

covariance

function

version

of

(9.65),

we

have

Cyy() =Cxx() h()h() =Cxx()Rhh(). (9.67)

h()h()=Rhh()

Note that thedeterministiccorrelation functionof h(t) is stillwhatweuse, evenwhenrelatingthecovariancesoftheinputandoutput. Onlythemeansoftheinputand output processes get adjusted in arriving at the present result; the impulseresponseisuntouched.

The correlation relations in Eqns. (9.59), (9.60), (9.63) and (9.65), as well astheir covariance counterparts, are very powerful, and we will make considerableuseofthem. Ofequal importancearetheirstatements intheFourierandLaplace

transform

domains.

Denoting

the

Fourier

and

Laplace

transforms

of

the

correlationfunction Rxx() by Sxx(j) and Sxx(s) respectively, and similarly for the other

correlationfunctionsofinterest,wehave:

Syx(j) =Sxx(j)H(j), Syy

(j) =Sxx(j)|H(j)|2,

Syx(s) =Sxx(s)H(s), Syy(s) =Sxx(s)H(s)H(s). (9.68)

WecandenotetheFourierandLaplacetransformsofthecovariancefunctionCxx()byDxx(j)andDxx(s)respectively,andsimilarlyfortheothercovariancefunctionsofinterest,andthenwritethesamesortsofrelationshipsasabove.

Exactlyparallel resultshold in theDTcase. Considera stablediscretetimeLTIsystemwhoseimpulseresponseish[n]andwhoseinputistheWSSrandomprocessx[n]. Then,asinthecontinuoustimecase,wecanconcludethattheoutputprocess

y[n]

is

jointly

WSS

with

the

input

process

x[n],

and

y

=x

h[n] (9.69)

Ryx[m] =h[m]Rxx[m] (9.70)

Ryy [m] =Rxx[m]Rhh[m], (9.71)

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


21/23

Section

9.8

The

Effect

of

LTI

Systems

on

WSS

Processes

181

whereRhh[m] isthedeterministicautocorrelationfunctionofh[m],definedas

+

Rhh[m] = h[n+m]h[n]. (9.72)n=

ThecorrespondingFourierandZtransformstatementsoftheserelationshipsare:

y

=H(ej0)x

, Syx(ej) =Sxx(e

j)H(ej), Syy(ej) =Sxx(e

j)|H(ej)|2,

y =H(1)x , Syx(z) =Sxx(z)H(z), Syy(z) =Sxx(z)H(z)H(1/z).(9.73)

Alloftheseexpressionscanalsoberewrittenforcovariancesandtheirtransforms.

Thebasicrelationshipsthatwehavedevelopedsofarinthischapterareextremelypowerful. InChapter10wewilluse these relationships to show that theFouriertransform of the autocorrelation function describes how the expected power of aWSS

process

is

distributed

in

frequency.

For

this

reason,

the

Fourier

transform

of

theautocorrelationfunctionistermedthepowerspectraldensity(PSD)oftheprocess.

TherelationshipsdevelopedinthischapterarealsoveryimportantinusingrandomprocessestomeasureoridentifytheimpulseresponseofanLTIsystem. Forexam-ple,from(9.70),iftheinputx[n]toaDTLTIsystemisaWSSrandomprocesswithautocorrelation function Rxx[m] = [m], then by measuring the crosscorrelationbetweenthe inputandoutputweobtainameasurementofthesystem impulsere-sponse. Itiseasytoconstructaninputprocesswithautocorrelationfunction[m],forexampleani.i.d. processthatisequallylikelytotakethevalues+1and1ateachtimeinstant.

As

another

example,

suppose

the

input

x(t)

to

a

CT

LTI

system

is

a

random

telegraph

wave,

with

changes

in

sign

at

times

that

correspond

to

the

arrivals

in

a

Poissonprocesswithrate, i.e.,

(T)keTP(kswitches inan intervalof lengthT) = . (9.74)

k!

Then,assumingx(0)takesthevalues1withequalprobabilities,wecandeterminethattheprocessx(t)haszeromeanandcorrelation functionRxx() =e

2||,soitisWSS(fort0). IfwedeterminethecrosscorrelationRyx()withtheoutputy(t)andthenusetherelation

Ryx() =Rxx()h(), (9.75)

wecanobtainthesystemimpulseresponseh(). Forexample,ifSyx(s), Sxx(s)and

H(s)

denote

the

associated

Laplace

transforms,

then

Syx(s)H(s) = . (9.76)

Sxx(s)

NotethatSxx(s)isaratherwellbehavedfunctionofthecomplexvariablesinthiscase, whereas any particular sample function of the process x(t) would not havesuchawellbehavedtransform. ThesamecommentappliestoSyx(s).

cAlan

V.

Oppenheim

and

George

C.

Verghese,

2010


22/23

182

Chapter

9

Random

Processes

As a third example, suppose that we know the autocorrelation function Rxx[m]of the input x[n] toaDTLTI system, butdonothaveaccess to x[n]and there-

fore

cannot

determine

the

crosscorrelation

Ryx[m]

with

the

output

y[n],

but

can

determinetheoutputautocorrelationRyy

[m]. Forexample,if

Rxx[m] =[m] (9.77)

andwedetermineRyy [m]tobeRyy [m] =

21|m|

,then1|m|

Ryy [m] = =Rhh[m] =h[m]h[m]. (9.78)2

Equivalently,H(z)H(z1)canbeobtainedfromtheZtransformSyy

(z)ofRyy

[m].Additionalassumptionsorconstraints, for instanceon the stabilityandcausalityof the system and its inverse, may allow one to recover H(z) from knowledge of

H(z)H(z1).

Alan

V.

Oppenheim

and

George

C.

Verghese,

2010

c


23/23

MIT OpenCourseWarehttp://ocw.mit.edu

6.011 Introduction to Communication, Control, and Signal Processing

Spring 2010

For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
http://ocw.mit.edu/http://ocw.mit.edu/termshttp://ocw.mit.edu/termshttp://ocw.mit.edu/

ramdom signals

Documents