Random Sequences

Gaurav S. Kasbekar

Dept. of Electrical Engineering

IIT Bombay

Recall

• We started with one r.v. 𝑋 on a probability space (Ω, ℱ, 𝑃)

• Then, two r.v.s 𝑋 and 𝑌 on a common probability space (Ω, ℱ, 𝑃)

• Then, a vector (𝑋1, … , 𝑋𝑛) of r.v.s on (Ω, ℱ, 𝑃)

• Next: an infinite sequence 𝑋1, 𝑋2, 𝑋3, … of r.v.s on (Ω, ℱ, 𝑃)

• We’ll study convergence of such a sequence

Motivation

• Two important results have to do with convergence of random sequences:

1) Law of Large Numbers

2) Central Limit Theorem

Law of Large Numbers • Recall motivation of the definition of 𝐸(𝑋)

• 𝑛 independent trials of experiment performed

• Average of values of 𝑋 in the 𝑛 trials used to motivate expression for 𝐸(𝑋)

• Let 𝑋1, 𝑋2, 𝑋3, … be i.i.d. with mean 𝜇

• lim𝑛→∞

𝑋1+⋯+𝑋𝑛

𝑛:

intuitively, 𝜇

• That is, letting 𝑋 𝑛 =𝑋1+⋯+𝑋𝑛

𝑛, the sequence

𝑋 1, 𝑋 2, 𝑋 3, … converges to the constant 𝜇

• To state this result formally, need to define convergence

Central Limit Theorem

• 𝑋1, 𝑋2, 𝑋3, … i.i.d. with mean 𝜇 and variance 𝜎2

• Informally, for large 𝑛, the CDF of 𝑋1 +⋯+ 𝑋𝑛 is approximately Gaussian

• That is, letting 𝑆𝑛 = 𝑋1 +⋯+ 𝑋𝑛, the distribution of 𝑆𝑛 converges to a Gaussian distribution as 𝑛 → ∞

Convergence of Real Numbers

• 𝑥1, 𝑥2, 𝑥3, …: a sequence of real numbers

• lim𝑛→∞

𝑥𝑛 = 𝑥 if:

for every 𝜖 > 0, there exists 𝑁𝜖 such that |𝑥𝑛 − 𝑥| < 𝜖 for all 𝑛 ≥ 𝑁𝜖

• E.g., limit of 𝑥𝑛 =(−1)𝑛

𝑛:

0

• E.g., limit of 𝑥𝑛 = (−1)𝑛:

does not exist (sequence oscillates)

Convergence of Random Variables

• 𝑋1, 𝑋2, 𝑋3, … r.v.s on (Ω, ℱ, 𝑃)

• Want to define convergence of this sequence

• Recall: 𝑋𝑖 is a function from Ω to ℛ

• So convergence of r.v.s similar to convergence of functions

• Simplest notion: point-wise convergence

called sure convergence in r.v. terminology

Sure Convergence

• Definition: 𝑋1, 𝑋2, 𝑋3, … converges surely to 𝑋 if for every ω ∈ Ω, lim

𝑛→∞𝑋𝑛(ω) = 𝑋(ω)

• E.g.: box initially has 𝑤 white and 𝑏 black balls

• At each step 𝑛 = 1,2,3, … one ball is drawn at random without replacement (if any left)

• 𝑋𝑛: number of white balls left after 𝑛’th draw

• Convergence behaviour of 𝑋1, 𝑋2, 𝑋3, … :

converges surely to 0

Example

• A fair coin tossed an infinite number of times

• 𝑋𝑛 = 1 if at least one of tosses 1,… , 𝑛 results in heads and 𝑋𝑛 = 0 else

• Convergence behaviour of 𝑋1, 𝑋2, 𝑋3, … :

with probability 1, converges to 1

but for ω = "𝑇𝑇𝑇… " ∈ Ω, 𝑋𝑛 ω = 0 for all 𝑛

does not converge surely to 1

• "𝑇𝑇𝑇… " is an event of probability 0

• Typically we don’t care about 0 probability events

Almost Sure Convergence

• Definition: 𝑋1, 𝑋2, 𝑋3, … converges almost surely to 𝑋 if for every ω ∈ 𝐴 ⊆ Ω, lim𝑛→∞

𝑋𝑛(ω) = 𝑋(ω), where P 𝐴 = 1

• In coin tossing example, 𝑋1, 𝑋2, 𝑋3, … converges a.s. to 1

Example

• Ω = 0,1 , ℱ = ℬ, 𝑃 𝑎, 𝑏 = 𝑏 − 𝑎, 0 ≤ 𝑎 ≤ 𝑏 ≤ 1

• 𝑋𝑛 ω = ω𝑛

• For fixed ω ∈ Ω, lim𝑛→∞

𝑋𝑛(ω) :

0, 0 ≤ ω < 11, ω = 1.

• 𝑋1, 𝑋2, 𝑋3, … converges a.s. to: 0

• Thus, one way to show a.s. convergence of 𝑋1, 𝑋2, 𝑋3, … to 𝑋:

identify 𝐴 such that lim𝑛→∞

𝑋𝑛(ω) = 𝑋(ω) for all ω ∈ 𝐴

show that P 𝐴 = 1

Almost Sure Convergence

• In several examples where intuitively 𝑋1, 𝑋2, 𝑋3, … seems to converge to 𝑋,

a.s. convergence does not hold

Example • Ω = 0,1 , ℱ = ℬ, 𝑃 𝑎, 𝑏 = 𝑏 − 𝑎, 0 ≤ 𝑎 ≤ 𝑏 ≤ 1

Ref: Hajek, Chapter 2

Example (contd.)

• lim𝑛→∞

𝑋𝑛(ω):

does not exist for any ω ∈ Ω!

• So 𝑋1, 𝑋2, 𝑋3, … does not converge a.s. to any r.v. 𝑋

• But intuitively, the sequence seems to be converging to 0