ivm dip apendix filtering
Post on 02-Jun-2018
233 Views
Preview:
TRANSCRIPT
-
8/10/2019 IVM DIP Apendix Filtering
1/48
Digital Image Processing
Hongkai XiongDepartment of Electronic Engineering
Shanghai Jiao Tong University
http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
2/48
Today
Wiener Filter
Kalman Filter Particle Filter
http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
3/48
Wiener Filter
In signal processing, theWiener filteris afilter proposed by Norbert Wiener during the1940s and published in 1949.
[1] Wiener, Norbert (1949).Extrapolation, Interpolation, andSmoothing of Stationary Time Series.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
4/48
The goal of the Wiener filter is to filter out noisethat has corrupted a signal. It is based on a
statistical approach. The design of the Wiener filter takes a different
approach. One is assumed to have knowledge ofthe spectral properties of the original signal andthe noise, and one seeks the linear time-invariant filter whose output would come asclose to the original signal as possible.
Wiener Filter Description
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
5/48
1. Assumption: signal and (additive) noise arestationary linear stochastic processes with
known spectral characteristics or knownautocorrelation and cross-correlation.
2. Requirement: the filter must be physicallyrealizable/causal.
3. Performance criterion: minimum mean-square error (MMSE)
Wiener Filter Description
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
6/48
Wiener Filter Problem Setup The input to the Wiener filter is assumed to be a
signal, , corrupted by additive noise, . The
output, , is calculated by means of a filter, ,using the following convolution:
is the original signal
is the noise
is the estimated signal
is the Wiener filters impulse response
)( ts
)(ts )(tn
)]()([*)()( tntstgts
where
)(ts
)(tn
)( ts
)(tg
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
7/48
Wiener Filter Problem Setup The error is defined as:
is the delay of the Wiener filter
In other words, the error is the difference between the estimatesignal and the true signal shifted by
)(
)()( tstste
where
)()()(2)()( 222 tstststste
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
8/48
Depending on the value of , the problem can bedescribed as follows:
1. If then the problem is that of prediction(error is reduced when similar to a later value ofs)
2. If then the problem is that of filtering
(error is reduced when similar to s)3. If then the problem is that of smoothing
(error is reduced when similar to an earlier valueof s)
0
0
0
)(
)()( tstste
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
9/48
)]()([*)()(
tntstgts
dtntsgts )]()()[()(
Writing as a convolution integral:
We denote:is the observed signal
is the autocorrelation function ofis the autocorrelation function ofis the cross-correlation function of and
)()()( tntstx
sR
xR
xsR
)(ts)(tx
)(ts )(tx
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
10/48
Taking the expected value of thesquared error results in
)(
)(
)(2)()( 222
tstststste
ddRggdRgReE xxss )()()()()(2)0()( 2
If the signal and the noise are uncorrelated (i.e., the cross-correlation iszero), then this means that
snR
sxs RR
nsx RRR
h
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
11/48
Wiener Filter Solutions
where
The Wiener filter problem has solutions for threepossible cases:1. The case where a non-causal filter is acceptable
(requiring an infinite amount of both past andfuture data)
2. The case where a causal filter is desired (usingan infinite amount of past data)3. The finite impulse response (FIR) case where a
finite amount of past data is used.
where
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
12/48
Wiener Filter Solutions
where
The first case (non-causal) is simple to solve but is notsuited for real-time applications.
Wiener's main accomplishment was solving the case wherethe causality requirement (second case) is in effect, and inan appendix of Wiener's book also gave the FIR solution.
Non-causal simple to solve, not suitable forapplication
Causal main accomplishmentFIR used in discrete series
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
13/48
Wiener Filter SolutionsNon-causal solution:
s
x
sx esSsSsG )()()( ,
dRgReE sxs )()()0()( ,
2
Provided that is optimal, then the minimum mean-square error equationreduces to
)(tg
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
14/48
Wiener Filter SolutionsCausal solution:
)(
)()(
sS
sHsG
x
where consists of the causal part of
is the causal component of (i.e., the inverse Laplace transformof is non-zero only for ) is the anti-causal component of (i.e., the inverse Laplacetransform of is non-zero only for )
)(sH s
x
sxe
sS
sS
)(
)(,
)(sSx
)(sSx
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
15/48
Wiener Filter SolutionsFinite Impulse Response Wiener filter fordiscrete series:
The causal finite impulse response (FIR) Wiener filter,instead of using some given data matrix X and outputvector Y, finds optimal tap weights by using the statisticsof the input and output signals
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
16/48
In order to derive the coefficients of the Wiener filter,consider the signal w[n] being fed to a Wiener filter oforderN. The output of the filter is denotedx[n] which isgiven by the expression
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/commons/0/00/Wiener_block.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
17/48
The residual error is denoted e[n] and is defined as
e[n] =x[n] s[n] (see corresponding block diagram).The Wiener filter is designed so as to minimize themean square error (MMSE criteria) which can bestated concisely as follows:
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
18/48
Assuming that w[n] and s[n] are each stationary andjointly stationary, the sequences known respectively asthe autocorrelation of w[n] and the cross-correlationbetween w[n] and s[n] can be defined as follows:
The derivative of the MSE may therefore be rewritten as
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
19/48
Letting the derivative be equal to zero results in
which can be rewritten in matrix form
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
20/48
Kalman Filter The Kalman filter, also known as linear quadratic
estimation(LQE), is an algorithm which uses a series
of measurements observed over time, containing noise(random variations) and other inaccuracies, andproduces estimates of unknown variables that tend to bemore precise than those that would be based on a single
measurement alone.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
21/48
Kalman Filter: OverviewThe Kalman filter uses a system's dynamics model (e.g.,physical laws of motion), known control inputs to that
system, and multiple sequential measurements (such asfrom sensors) to form an estimate of the system'svarying quantities (its state) that is better than theestimate obtained by using any one measurement alone.As such, it is a common sensor fusion and data fusionalgorithm.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
22/48
Kalman Filter: Application The Kalman filter is an efficient recursive filter that
estimates the internal state of a linear dynamic system
from a series of noisy measurements. It is used in a widerange of engineering and econometric applications fromradar and computer vision to estimation of structuralmacroeconomic models, and is an important topic incontrol theory and control systems engineering.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
23/48
Kalman Filter: dynamic system model
Fk, the state-transition model;
Hk, the observation model;Qk, the covariance of the process noise;Rk, the covariance of the observation noise;Bk, the control-input model.
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
24/48
Kalman Filter: dynamic system model
Fkis the state transition model which is applied to the previous state xk1;
Bkis the control-input model which is applied to the control vector uk;
wkis the process noise which is assumed to be drawn from a zero meanmultivariate normal distribution with covariance Qk.
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
25/48
Kalman Filter: dynamic system model
Hkis the observation model which maps the true state space into theobserved space andvkis the observation noise which is assumed to be zeromean Gaussian white noise with covariance Rk.
,
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
26/48
Kalman Filter: dynamic system model
The Kalman filter is a recursive estimator. This means thatonly the estimated state from the previous time step and
the current measurement are needed to compute theestimate for the current state.
The state of the filter is represented by two variables:
, theposterioristate estimate at time kgivenobservations up to and including at time k;
, theposteriorierror covariance matrix (a measureof the estimated accuracy of the state estimate).
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
27/48
Kalman Filter: Predict
Predicted state estimate:
Predicted estimate covariance:
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
28/48
Kalman Filter: Update
Innovation or measurement residual
Innovation (or residual) covariance
OptimalKalman gain
Updated state estimate
Updated estimate covariance
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
29/48
Kalman Filter: Invariants If the model is accurate, and the values for andaccurately reflect the distribution of the initial state values,then the following invariants are preserved: (all estimates
have a mean error of zero)
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
30/48
Kalman Filter: Estimation of the noise
covariances Qkand Rk
Practical implementation of the Kalman Filter is oftendifficult due to the inability in getting a good estimate of
the noise covariance matrices Qkand Rk. Extensiveresearch has been done in this field to estimate thesecovariances from data. One of the more promisingapproaches to doing this is called theAuto-covarianceLeast-Squares (ALS)technique that uses
autocovariances of routine operating data to estimate thecovariances.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
31/48
Kalman Filter: An example applicationConsidering a model:
A truck on infinitely long straight rails. Initially, the truck is stationary atposition 0, but it is buffeted this way and that by random acceleration. Wemeasure the position of the truck every tseconds, but these
measurements are imprecise; we want to maintain a model of where thetruck is and what its velocity is.
The position and velocity of the truck are described by the linear state space
We show here how we derive the model from which we createour Kalman filter:
is the velocity, that is, the derivativeof position with respect to time.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
32/48
Kalman Filter: An example application
We assume that between the (k 1) and ktimestep the truckundergoes a constant acceleration of akthat is normally distributed,with mean 0 and standard deviation a. From Newton's laws ofmotion we conclude that
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
33/48
Kalman Filter: An example application
At each time step, a noisy measurement of the true position of thetruck is made. Let us suppose the measurement noise vkis alsonormally distributed, with mean 0 and standard deviation z.
We know the initial starting stateof the truck with perfect precision,
so we initialize
To tell the filter that we knowthe exact position, we give it a
zero covariance matrix:
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
34/48
Particle Filter: In statistics, a particle filter, also known as a sequential
Monte Carlomethod (SMC), is a sophisticated modelestimation technique based on simulation.[1] Particle
filters are usually used to estimate Bayesian models inwhich the latent variables are connected in a Markov chainsimilar to a hidden Markov model (HMM), but typicallywhere the state space of the latent variables is continuousrather than discrete, and not sufficiently restricted to makeexact inference tractable
[1] Doucet, A.; De Freitas, N.; Gordon, N.J. (2001).Sequential Monte Carlo Methods in Practice. Springer
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
35/48
Particle Filter: Particle filters are the sequential analogue of Markov chain
Monte Carlo (MCMC) batch methods and are often similarto importance sampling methods.
Adventages:Well-designed particle filters can often be much faster thanMCMC.With sufficient samples, they approach the Bayesian
optimal estimate, so they can be made more accurate thaneither the EKF or UKF.Disadvantages:
When the simulated sample is not sufficiently large, theymight suffer from sample impoverishment.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
36/48
Particle Filter: GoalThe particle filter aims to estimate the sequenceof hidden parameters,xkfor k= 0,1,2,3,, based
only on the observed data ykfor k= 0,1,2,3,.
All Bayesian estimates ofxkfollow from theposterior distributionp(xk| y0,y1,,yk). In
contrast, the MCMC(Markov chain Monte Carlo)or importance sampling approach would modelthe full posteriorp(x0,x1,,xk| y0,y1,,yk).
.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
37/48
Particle Filter: ModelParticle methods assume and the observations can bemodeled in this form:
is a first order Markov process such that
and with an initial distribution .
The observations are conditionally independentprovided that are knownIn other words, each only depends on
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
38/48
Particle Filter: ModelOne example form of this scenario is
where both and are mutually independent and identicallydistributed sequences with known probability density functionsand and are known functions.
These two equations can be viewed as state space equations andlook similar to the state space equations for the Kalman filter.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
39/48
Particle Filter: Monte Carlo approximation
Particle methods, like all sampling-based approaches,generate a set of samples that approximate the filtering
distribution
So, with P samples, expectations with respect to thefiltering distribution are approximated by
.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
40/48
Particle Filter: Sequential ImportanceResampling (SIR)
Sequential importance resampling (SIR), the originalparticle filtering algorithm, is a very commonly used
particle filtering algorithm, which approximates thefiltering distribution by a weighted set of Pparticles
The importance weights are approximations to therelative posterior probabilities (or densities) of theparticles such that .
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
41/48
Particle Filter: Sequential ImportanceResampling (SIR)
SIR is a sequential version of importance sampling. As in
importance sampling, the expectation of a functioncan be approximated as a weighted average
.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
42/48
Particle Filter: Sequential ImportanceResampling (SIR)For a finite set of particles, the algorithm performance isdependent on the choice of theproposal distribution
The optimal proposal distributionis given as the targetdistribution
However, the transition prior is often used as importancefunction, since it is easier to draw particles (or samples) andperform subsequent importance weight calculations:
,
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
43/48
Particle Filter: Sequential ImportanceResampling (SIR)A single step of sequential importance resampling is asfollows:
1) For draw samples from theproposaldistribution
2) For update the importance weights up to anormalizing constant:
Note that when we use the transition prior as theimportance function,this simplifies to the following:
,
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
44/48
Particle Filter: Sequential ImportanceResampling (SIR)
3) For compute the normalized importance weights:
4) Compute an estimate of the effective number of particles as
5) If the effective number of particles is less than a given threshold ,then perform resampling:a) Draw P particles from the current particle set with probabilities
proportional to their weights. Replace the current particle set with thisnew one.
b) For , set
,
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
45/48
Particle Filter:
Result of particle filtering (red line) based on observed data generatedfrom the blue line
,
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/commons/5/5f/Particle_filter_example_small.pnghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
46/48
Particle Filter: "Direct version" algorithm
,
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
47/48
Other Particle Filter:
Auxiliary particle filter
Gaussian particle filterUnscented particle filterGauss-Hermite particle filterCost Reference particle filterRao-Blackwellized particle filterRejection-sampling based optimal particle filter
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/ -
8/10/2019 IVM DIP Apendix Filtering
48/48
Thank You!
http://ivm.sjtu.edu.cn/
top related