exact and stable covariance estimation from quadratic …...exact and stable covariance estimation...

Exact and Stable Covariance Estimation from Quadratic Sampling

Yuxin Chen? Yuejie Chi† Andrea J. Goldsmith?? Electrical Engineering, Stanford University † Electrical and Computer Engineering, Ohio State University

• Covariance Estimationin – second-order statistics Σ ∈ Rn×nin – cornerstone of many information processing tasks

• Quadratic Samplingin – obtain measurements of the form

y ≈ a>Σa (1)

Objectives

binary data stream by Kazmin

• Covariance Sketchingin – data stream: real-time data {xt}∞t=1 arrivingsequentially at a high rate...

• Challengesin – limited memoryin – computational efficiencyin – a single pass over the data

Application: Data Stream Processing

• Proposed qudratic sketching method!  Quadratic)Sketching)Scheme!

!x1"

!x2"

!x3"

!x4"

!x5"

!x6"

!x7"

!xN!

a1T"

!xi!

amT"

random))quadratic)sampling:)"a1T xixi

Ta1

xiT" !a1"

y1 =1N

a1T xixi

Ta1i=1

N

∑

aggregate)all)sketches"

N"data"instances"!"m"sketches"

random)sampling)"

n6dim)

1. sketching:

– at each time t, randomly choose a sketching vector ai– observe a quadratic sketch (aTi xt)

2

!  Quadratic)Sketching)Scheme!

!x1"

!x2"

!x3"

!x4"

!x5"

!x6"

!x7"

!xN!

a1T"

!xi!

amT"

random))quadratic)sampling:)"a1T xixi

Ta1

xiT" !a1"

y1 =1N

a1T xixi

Ta1i=1

N

∑

aggregate)all)sketches"

N"data"instances"!"m"sketches"

n6dim)

2. aggregation:

– all sketches are aggregated into m measurements

yi ≈ E(a>i xtx

>t ai

)= a>i Σai (1 ≤ i ≤ m)

3. goal (estimation)

– estimate the covariance matrix Σ from y := {yi}mi=1

• Benefits

– one pass

– minimal storage (as will be shown)

Sketching*

Aggrega.on*

Es.ma.on*

• High-frequency wireless and signal processing

– Stationary processes / signals (possibly sparse)

t1 t2 t3– In high-frequency regime, energy measurements are more reliable

– Goal: recovery the power spectral density from energy measurements

∗ Frequency spikes can assume any continuous value (off-the-grid).

Application: Spectral Estimation

• Suppose that rank(Σ) = r � n

The Task

C = A* B*+

Low-rank Matrix

Unknown rank, eigenvectors

Sparse “Errors” Matrix

Unknown support, values

GivenComposite

matrix

• Convex Relaxation (TraceMin)

minimize trace (M )

s.t. ‖A (M )− y‖1 ≤ ε1,

M � 0.

– This coincides with PhaseLift (Candes et. al.) forrank-1 cases.

Formulation

• Given m (� n2) quadratic measurements y = {yi}mi=1

yi = a>i Σai + ηi, i = 1, · · · ,m, (2)

we wish to recover Σ ∈ Rn×n.

– ai : sketching/sampling vectors

– η = {ηi}mi=1: inaccuracy / noise terms

– More concise operator form: y = A(Σ) + η

• Sampling model

– sub-Gaussian i.i.d. sketching / sampling vectors

Algorithm: general low-rank structure

• Convex Relaxation (ToeplitzMin)

minimize M1,1

s.t. ‖A (M )− y‖1 ≤ ‖η‖1 ,M � 0,

M is Toeplitz.

– Can acommodate off-grid frequencies.

Algorithm: spectrally sparse processes

Piet Mondrian

Theoretical Guarantee

Theorem. (Non-stationary) If ‖η‖1 ≤ ε1, then the solution to TraceMin obeys∥∥∥Σ̂−Σ∥∥∥

F≤ C1

‖Σ−Σr‖∗√r

+ C2ε

m, (3)

where Σr is the best rank-r approximation of Σ, provided that m > Θ(nr).

(Stationary) If ‖η‖2 ≤ ε2, then the solution to ToeplitzMin obeys∥∥∥Σ̂−Σ∥∥∥

F≤ C3ε2√

m, (4)

provided that rank(Σ) ≤ r and m > Θ(r · poly log(n)).

m / (n*n)

r/n

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

information theoretic limit

m: number of measurements

r: r

ank

5 10 15 20 25 30 35 40 45 500

5

10

15

20

25

30

35

40

45

50

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

theoretic sampling limit

(Left) Non-stationary case; (Right) Stationary case.

• Exact and Universal Recovery: from minimal noiseless measurements;

• Stable Recovery: inaccuracy proportional to noise level;

• Robust Recovery: robust to imperfect structural assumptions.

Discussion

• Work for other structural models

– Sparse covariance matrix

– Sparse phase retrieval (rank-1)

– Simultaneous sparse and low-rank covariance model

• Paper: Exact and Stable Covariance Estimation from Quadratic Samplingvia Convex Programming (http://arxiv.org/abs/1310.0807)