1
NMIJ - BIPM workshop on theImpact of Information Technology in Metrology
18 -20 May 2005, AIST Tsukuba, Japan
Forward calculation
Cause EffectCause Effect
Inverse problem
Model
Measurand ObservablesMeasurand Observables
SampleSampleGas target
xx
zz
D(d,n) X(n,n')X(n,n')
DetectorDetector
ϑn,n'ϑn,n'
Simulation („virtual experiment“) 0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
Measurement 0 0
5 5
1010
1515kgkg
MeasurementMM
M MM
Evaluation of uncertainty
The use of Monte Carlo techniques for metrology- evaluation of uncertainties and simulation of experiments –
Bernd R.L. Siebert*, Dieter Sibold* and Klaus-Dieter Sommer+
* Physikalisch-Technische Bundesanstalt in Braunschweig und Berlin (PTB)+ Landesamt für Mess- und Eichwesen Thüringen (LMET)
The presentation does not discuss new results, but is to demonstrate thato Monte Carlo integration allows to treat any physically sound
model for the evaluation of uncertainty and thereby providesadded value to the Guide on the Expression of Uncertainty inMeasurement (GUM) as it is not restricted to linear models and
o Monte Carlo simulation is a versatile tool to analyse and evaluatecomplex measurements.
MC integration and MC simulation are not generally used terms. However, it can be shown that uncertainty evaluation using given models according to GUM (Guide for the expression of uncertainty in measurement) can be transformed to a simple integration problem whereas “virtual” experiments often require a “translation” of the model into a step-by-step procedure, especially so if the underlying physics is described by (a system of) integral equations. The term “simulate” means then to “mock-up” cause-and-effect chains by a chain of physical interactions. Here, the model itself is also under study.Two simple concurrent examples serve to demonstrate this.
2
2The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Content
IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
1 Summary and Conclusions5 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
3 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
24 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
Main message:Inference from given data (effects) to possible causesis a major task of metrology,Monte Carlo techniques provideversatile, powerful and easy to handle toolsto that end. Introduction
Cause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
1 IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
11
Summary and Conclusions5 Summary and Conclusions55
Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
3 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
33
Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
2 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
22
4 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
44 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
Complete result of ameasurement (calculation):y & u(y) or y & U.Model is given.
Interprets, supplements oreven replaces an experiment.Model is under study!
Content
IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
1 IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
11 Summary and Conclusions5 Summary and Conclusions55 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
3 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
33 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
2 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
224 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
44 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
Main message:Inference from given data (effects) to possible causesis a major task of metrology,Monte Carlo techniques provideversatile, powerful and easy to handle toolsto that end. Introduction
Cause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
1 IntroductionCause-and-Effect Chain and Model, Integration ↔ SimulationDeterministic and Stochastic Models
11
Summary and Conclusions5 Summary and Conclusions55
Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
3 Evaluation of UncertaintyGUM concept, Markov Formula & Standard Procedure
33
Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
2 Monte CarloIntroduction, Random numbers, Sampling, Sorting & „Statistics“
22
4 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
44 Simulation („virtual experiment“)Model, Inference, “Validation” & Uncertainty
Complete result of ameasurement (calculation):y & u(y) or y & U.Model is given.
Interprets, supplements oreven replaces an experiment.Model is under study!
We use two concurrent examples:o Measurement of a mass with or calibration of a spring scale to
demonstrate Monte Carlo integration ando Determination of a differential neutron cross section with a
time-of-flight experiment to demonstrate Monte Carlo simulation.
Clearly, simple spring scales are not used much in practice, but theyare ideal for demonstration, as they require, although simplified, tomodel effects that are of importance with many types of
measurements.
The time-of-flight experiment is still running at PTB. The material used here serves for demonstration only and it is taken from a MonteCarlo code developed some years ago (1975-1995) for theanalysis of this experiment. This problem was selected since itexemplifies an evaluation of a measurement that does require a very detailed simulation of a long cause-and-effect (Markov) chains. References see bottom of viewgraph 19.
PDF-Dateiöffnen
We use two concurrent examples:o Measurement of a mass with or calibration of a spring scale todemonstrate Monte Carlo integration and
o Determination of a differential neutron cross section with atime-of-flight experiment to demonstrate Monte Carlo simulation.
Clearly, simple spring scales are not used much in practice, but theyare ideal for demonstration, as they require, although simplified, tomodel effects that are of importance with many types of measurements.
The time-of-flight experiment is still running at PTB. The material used here serves for demonstration only and it is taken from a MonteCarlo code developed some years ago (1975-1995) for theanalysis of this experiment. This problem was selected since itexemplifies an evaluation of a measurement that does require a very detailed simulation of a long cause-and-effect (Markov) chains.
3
3The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Introduction Cause-and-Effect Chain and Model (1/4)
1
Cause Effect
Measurand Observables
Yes, but
0 0
5 5
1010
1515kgkgM
0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
MeasurementM M
Can one infer the valuem of the mass M andu(m) from mIND= 5 kg?
one needs a model,to link the measurandand the observed data
SampleSampleGas target
xx
zz
D(d,n) X(n,n')X(n,n')
DetectorDetector
ϑn,n'ϑn,n'
Cyc
lotro
n:Em
ittan
ce
Time of flightspectra
Can one infer thedifferential crosssection value σ (ϑn,n’)from TOF-spectra?
Inference requires a model
We use the term model for a description of a cause action chain asconstrued for a defined purpose.
The purpose is to predict a consequent (posterior) state Xpost
if one is given the antecedent (prior) state Xante.A model is not “true” but “reasonable” or “believable”. The verb “construe” is to mean: deduce by inference from given information, i.e. data in the widest meaning. For instance, the fact that many physical relations are locally linear is a datum. Basis for construing a model can be a theory or a phenomenological relation, validated by experiments.
NOTE 1: A given effect (observed data) can result from different causes, e.g. thelength of a spring is affected by an applied force or a change intemperature. A cause, e.g. “a neutron passes through matter”, can resultin a distribution of effects, here the path length to the location of thenext interaction, that may be interpreted as PDF.
NOTE 2: Occam's razor* should be used for guidance, e.g. separate linear modelsare sufficient for small changes of Xante , i.e. ∆F and ∆T.*Here: The simplest explanation that fits all data given is the best one.
4
4The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Introduction Integration ↔ Simulation (2/4)
1
A given effect canresult from differentcauses,
Inference from effect is an an inverse problem that may be ill-posed.
Cause Effect
Measurand Observables
Cause Effect
Measurand Observables
Cause Effect
Measurand Observables
A model allows a forward calculation from cause to effect.
e.g. a decrease in temperatureor an increase of exerted forcedecrease the length of a spring.
A given cause canresult in differentobservable effects,
e.g. a neutron in matter:the path length between tworeactions is distributed.
Forward calculation
Inverse problem
Model
The model for the evaluation of uncertainty allows to infer the value m of themass M and u(m) from mIND= 5 kg via forward calculation, here integration.
The model for the simulation of the time-of-flight experiment (TOF) allows toinfer σ (ϑn,n’) from TOF-spectra via forward calculation, here simulation.
The term model is here used as introduced on viewgraph 3. However,a cause-and-effect chain may be very long. In this case we use sub-models to describe effects due to “separable” causes. This separation is helpful for the later computation, but one encounters two problems:o generally, it is impossible to identify all possible causes ando the description of the identified cause-and-effect chain is often only
an approximation, either for lack of knowledge or due to mathe-matical or computational problems in formulating them exactly.
See also the citation on viewgraph 6 on deterministic and stochastic models.For more clarity, we use the termo Model for evaluation of uncertainty for the relation that allows to
infer from a given set of values of the input quantities determini-stically the value of the output quantity (or values of the output quantities) (this leads to an integration problem) and
o Model for simulation for the set of steps or procedures and theparameters used to realise a Markov chain for the entity understudy, here, in the concurrent example, deuterons and neutrons.NOTE: Not all simulation problems lead to Markov chains. For example, the
use of point detectors in transport calculations may violate Markovconditions if used not only for immediate tallying.
5
5The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
0 0
5 5
1010
1515kgkg
M
Introduction Deterministic model (3/4)
Example: change of the length of a spring:
lspring lengthXpost
∆Traise temperature
∆Fapply force
l0spring lengthXante
( )Tll T∆10 α+=
( )Fll F∆10 α+= ( )( )TFll TF ∆1∆10 αα ++=
Simplification for small values of
∆F and ∆T !
1
Basis: Hooke‘s law and temperature expansion
( )TllT,lT ∂
∂= −1
00α( )FllF,lF ∂
∂= −1
00α
The model allows to “propagate” the PDFs for the force F and the temperature T. Example: The information on temperature can be simple, e.g. one knows that the values obey the relation tmin < t <tmax → via PME gT(t) rectangular. Alas, it may also be appropriate to assign a U-shaped PDF to account for sinusoidal control.
-continued form notices on bottom of page 4 –Aristotle, if he lived today, might say:The purpose of the model is its causa finalis for making it and the model itself is an algorithms that reflects our knowledge of the causa efficiensunder study. He might add that thoughts are the causa materialis, and he might identify the algorithm as causa formalis.----------------------------------------------------------------------------------Although Hooke’s law and temperature expansion can be understood microscopically, they are in practice merely used as phenomenological relation with experimentally determined values of parameters, usually termed “coefficients”, i.e. thermal length (or volume) expansion coefficient.This is representative of a widely used modelling technique: From experiments, or, if available, preferentially from theories, one expects a relation and determines parameters that cannot be computed (either not at all or with undue expenditure only) by measurements.On viewgraph 15 is a more correct model for treating the effect of exerting a force. However, this model is already so “complex” that approximations are needed to obtain solutions with “justifiable expenditure. However, what is “justified” is determined be the aimed at smallest uncertainty contribution.
6
6The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Introduction Stochastic model (4/4)
1
One obtains a PDF for path length to next reaction:
( ) ( )[ ] ( )mattern
matternexp
ntot
ntot
,,E,,E-
g X ΣξΣ
ξ =
Example: path length between two reactions of a neutron in matter:Basis: Attenuation of neutron fluence in matter
( ) ( ) ( )( )[ ]0ntotn0n matternexpnn ξξΣξΦξΦ −= ,,E-,E,,E,
n: spin, magnetic field,... matter: nuclei, temperature,
cross sections ...
internal states ?
Whether or not a stochastic model could be transformed to a deterministic oneif one were to know all internal states or so-called hidden variables may be disputed,in practice, however, the information given corresponds to incomplete knowledge!
Therefore, we consider as a PDF that reflects our (incomplete) knowledgeabout the quantity path length between two reactions of a neutron in matter.
( )ξXg
The prior knowledge about the interaction site is expressed by a PDF,but there will be exactly one site where the next interaction occurs!
Stochastic models can be used with Monte Carlo codes and deterministic codes. See for example: Hamid Tagziria, H. (editor): Review of Monte Carlo and Deterministic Codes in Radiation Protection and Dosimetry, NPL-report, Teddington, UK, 2000, ISBN: 0 946754 32. We cite from the introduction of that report:Modelling a physical system can be carried out either stochastically or deterministically. An example of the former method is the Monte Carlo technique in which statistically approximate methods are applied to exact models. No transport equation is solved as individual particles are simulated and some specific aspect (tally) of their average behaviour is recorded. The average behaviour of the physical system is then inferred using the central limit theorem. In contrast, deterministic codes use mathematically exact methods that are applied to approximate models to solve the transport equation for the average particle behaviour. The physical system is subdivided in boxes in the phase-space system and particles are followed from one box to the next. The smaller the boxes the better the approximations become.Note on sub-models: It can be shown, that most models can be broken down into few types of sub-models. Sub-models are easier to test and they simplify the encoding of models in higher computing languages, such as FORTRAN.
7
7The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Step by step simulation of possible paths.
Monte Carlo Method Introduction (1/7)
1 2
Law of Large Numbers: If the expectation value EX exists, then for any εE > 0 exists a M < ∞.
( ) ( ) E1
1-max
min
d E ε+⋅== ∑∫ mm
M
m=gMVg ξξξ ξξξ
x
x
( ) ( ) (volume) minmaxT
minmax xxxx −−=V
Why Monte Carlo? Analytical or othernumerical methods
Validation?
Convergence?
If the variance exists, then one can compute u( EX )
[ ] ( )( )∫ −=max
min
d EVar 2x
x
ξ ξξξξ g
Variance:
• Integrals• Integral equations• Random walk• Bootstrap• ........xmin, xmax , ξ and ξm are vectors of dimension N, M denote the number of samples and
m a sample, the nth element of ξm is given by ξm,n = (xmax,n − xmin,n ) ρm,n and ρm,n is arandom number, ρm,n ∈[0,1].
The Monte Carlo method (MCM) is concerned with the application of random sampling to problems of applied mathematics. A short and well-coined description has been given by H. Kahn in. Applications of Monte Carlo, Report AECU-3259 (U.S. Atomic Energy Commission, The Rand Corporation, Santa Monica, CF /USA), 1956) :The expected score of a player in any reasonable game of chance,however complicated, can in principle be estimated by averaging the results of a large number of plays of the game. Such estimation can be rendered more efficient by various devices, which replace the original game with another known to have the same expected score. The new game may lead to a more efficient estimate by being less erratic, that is, having a score of lower variance or by being cheaper to play with the equipment at hand.The MCM is used in a wide range of applications in physics, e.g. Binder, K. (editor): Monte Carlo Methods in Statistical Physics. Topics in Current Physics, Vol. 7 [36], (Berlin Heidelberg New York 1979 [1984], ISBN 3-540-09018-5 [3-540-12764-X].The internet provides ample information on history, theory and present applications of MCM to natural and other sciences. In this section we discuss only general principles. The concurrent ex-amples are discussed in section 3 (integration) and 4 (simulation).
8
8The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Monte Carlo Method Random numbers (2/7)
0 20 40 60 80 100 120 140ξ
10-9
10-8
10-7
10-6
10-5
10-4
10-3
10-2
G(ξ
)
TheoryTheory 2.1092.109
2.1062.106
16 digits are given16 digits are given
ξ = sum over their valuesξ = sum over their values
For both, integration and simulation,one needs many random numbers.(About 500 to thermalise a fast neutron)One is to rely on random numbersprovided by an algorithm. It is notpossible to verify such random numbergenerators!But one can validate them bycomputing problems for which oneknows the answer exactly.For instance, the frequency distribution of the sum over all digits.
1 2
If one has a random number generator that producesuniformly distributed values in the interval [0,1]one can reproduce any probability distribution function,be it continuous or discrete.
For instance, we use the portable random number generator that is suggested in the supplement* in a 16 digit representation and tested the generator by computing:o the frequency distribution of the sum over all digits (see Figure),o the frequency that the number 0,… or 9 occurs in a random once, twice …16 times,o the frequency that the number 0,… or 9 occurs in any of the 16 digits ando the frequency of chains of the numbers 0,… or 9 of length n =1,2,…8 starting at
the (17 - n)th-digit.For the four questions one can compute the exact answers. Furthermore, we tested for correlation and computed analytically known expectation values, variances and higher moments. For all values computed, we calculated in addition their variance by repeating the calculations many times. The tests did not exhibit any significant deviation from the expectations based on analytical calculations or logical reasoning. Therefore, we assume that the random generator used is sufficiently ideal for our applications; however, this is not a proof!
* See paper WS-35 by M. G. Cox, P. M. Harris and I. M. Smith:Supplements to the guide to the expression of uncertainty. in measurement.
9
9The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Monte Carlo Method Sampling(1/2) (3/7)
( )( ) x.xu +−= 503 ρξ
If a reliable random number generator is given one obtains:
a sample from an uniform (rectangular) PDF from
where, using GUM nomenclature, x is the expectation value and u(x)=σa uniform PDF is given if one only knows the limits x− and x+of the values that might be reasonably attributed to that quantity:(Theory: Principle of Maximum (information) Entropy.
Hand waving: Any other shape would pretend not given information!)
a sample from a triangular PDF can be obtained from a sum of two samplesfrom uniform PDFs of equal half width,a sample from a trapezoidal PDF from a sum of two samples fromuniform PDFs of different half width anda sample from a U-shaped PDF is obtained from
( )
( ) ( )xuxxa
xxx
32121
=−=
+=
−+
−+
( )( ) ( )xu. 250sin −= ρπξ
1 2
To simulate throwing a fair dice with n faces, labeled 1,...i,..none would use: , but use n if n+1 is sampled![ ] 1 += ρni
σ µ
The central limit theorem [(A.M. Ljapunoff (1857 - 1918)]. is used for two entirely different aspects of the problems studied here.(1) The PDF for an output quantity that is linearly combined from many input quantities tends to be a Gaussian according to the central limit theorem. Indeed, for instance the linear combination of a few rectangular PDFs with approximately the same variances can in practice not be dis-tinguished from a Gaussian. This is the basis for the often made statement“for a linear model with many input quantities one obtains a GaussianPDF for the measurand.”
(2) The distribution Z of repeatedly computed expectation values of any PDF Y tends to be a Gaussian if the number of samples, M, used for computing the expectation values becomes large. The variance of Z is according to the central limit theorem given by M−1 Var[Y]. If Y is a Gaussian then Z is a t-distribution of degree of freedom ν = M−1:
If Y is not a Gaussian one can, if needed, compute Z by Monte Carlo techniques.
( ) ( )xux
ν
NNg ,T−
=)2Γ(π
⎟⎠⎞
⎜⎝⎛ +
Γ=⎥
⎦
⎤⎢⎣
⎡+=
+−
ξτνννττ ν
ν
νν and 21
where12
12
10
10The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Monte Carlo Method Sampling (2/2) (4/7)
1 2
There are various sampling techniques. Analogue sampling works always:
-3 -2 -1 0 1 2 3ξ / u(x)
0
0.25
0.5
0.75
1
G (ξ
)
Cumulative distribution function1st step: G(ξ) = ρ
2nd step: ξ = G-1(ρ )
3rd step: ξ i,s = ξi u(xi)+ x
( ) ( )
( ) ( )
( ) ρξξξρξζζ
ξ
ρξζζ
ξ
ξ
ddddd
dd
d
=⇒
==
==
∫
∫
∞−
∞−
X
XX
XX
g
gg
Gg
If G(ξ) can not be expressed ana-lytically, one can use a discrete table: ⇒dice with n faces+ interpolation.
For many PDFs, e.g. a Gaussian or a t-distribution algorithms exist that produce a sample, e.g. based on rejection technique.
Correlation: Joint PDFs Choleskyfactorisation orauxiliary quantities.
Variance reduction: StratifiedImportance..........
Annotations:
Use standard forms, σ =1 and EX =0 ⇒ ξs= u(x) ξs,n + x
- continued from notices on bottom of page 9-In computing integrals and in doing step by step simulations we have the situation as described under (2) above, i.e. we can consider a computed expectation value. This allows then the simple statement: If the variance exists one can reach convergence and the uncertainty associated with the once computed expectation value of Y , u2(EYcomputed) = M−1 Var[Ycomputed].-----------------------------------------------------------------------------------------------------------------------------
Rejection technique (von Neumann):Consider a probability density function, g(ξ ), which can be surrounded by a box (or any function come to that) between xmin and xmax and havingheight gmax.The following algorithm yields - for a large sample number- g(ξ):(1) Create ξ = xmin+ (xmax − xmin ) ρ1
(2) Create g = gmax. ρ2
(3) If g < g(ξ), then accept ξ. If not, reject and go to (1).For transformation method see next page and for correlation and variance reduction see bottom of pages 12 and 13, respectively.
11
11The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Monte Carlo Method Sorting & statistics (1/3) (5/7)
1 2
Evaluation of uncertainty: Generation of a possible value of the measurand.
Model for the evaluation, Y=f(X1,...,XN), combines possible values of the Xi tothe possible values of the measurand: ( ) sample.current ; 1 == m,...,f m,Nm,m ξξη
We do not need to consider εE and εV since we can use the central limit theorem:
( ) .MyMMuM
mm
M
mm ∞→−⇒⎟⎠
⎞⎜⎝
⎛ ∑∑=
−
=
−
1
22
1
12 for ηηThe ηm can be sorted, this yieldsa frequency distribution thatapproximates the PDF for Y.Details see Section 3.
Caveat: Check conditions for central limit theorem.
This procedure is equivalent to solving integrals (see Sect. 3):
NNN ξξξfξg...y ...dd ),...,(),...,( 111 ξξ∫∫∞
∞−
∞
∞−
= X ( ) ( )( ) .ξξy,...,ξξfξg...yu NNN ...dd),...,( 12
112 −= ∫∫
∞
∞−
∞
∞−
ξX
( ) [ ] ( )∑∑=
−
=
− +−−==+==M
mm
M
mm yMYyuMYy
1Var
212E
1
1 .1)(Var and E εηεηFor large samplenumber M
-continued from notices on bottom of page 10 -Transformation methodConsider the general transformation: and introduce:
The Jacobian determinant can be computed and one obtains:
Take gR (ρ) is uniform for both, ρ1 and ρ1, one obtains 2 independent Gaussians in standard from this transformation. This is the Box-Muller method (Ann. Math. Staist.,29 (1958) p. 610.611.). It requires the calculation of sinus and cosines functions. This can be avoided by using rejection techniques, for detail see Press, W.H., Flannery, B.P., Teukolsky, S.A. and Vetterling, W.T.: Numerical Recipes -The Art of Scientific Computing-. Cambridge, Cambridge University Press (1989)).Correlation and variance reduction see next pages, 12 and 13,respectively.
( ) ( ) ( ) ( )and 2sinlog2 and 2coslog2 21e221e1 πρρξπρρξ −=−=
( ) ( ) ( )( ) ξρξ RX d
21
21
ξξρρ
,,gg
∂∂
=
( ) .- ⎟⎟⎠
⎞⎜⎜⎝
⎛=⎥⎦
⎤⎢⎣⎡ +=
2
12
22
211 acrtan
21 and
21exp
ξξ
πρξξρ
( )( ) .
,,
⎥⎦⎤
⎢⎣⎡−⎥⎦
⎤⎢⎣⎡−=
∂∂
2exp21
2exp21 2
22
1
21
21 ξπ
ξπξξ
ρρ
12
12The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Monte Carlo Method Sorting & Statistics (2/3) (6/7)
1 2
Step by step simulation: Generation of a history (1/2).
Step by step procedure yields a possible history of the object under study.
Random walk problem: Transition probabilities from one step to the next one.(interaction probability).
Associate a state vector with object under study.Number of elements depends on the purpose of the calculation.
Concurrent example:probability, time type of particles, energy, location, direction of flight and flags.
k is e.g. deuteron or neutron, flags indicate history
and via r the material and geometry are known. ( )flags,,,E,k,t,p Ωrs =T
Markov chain: Transition probabilities from a previous state to next statedepend only on the previous state vector.
E.g. D(d,n) happened in r ±∆ r and n was x-times scattered and ...
Properties of spectators⇒ modelling, optimising
“possible” ≡ (synonym) reasonably expected due to my (incomplete) knowledge.
CorrelationIn the case that the information about correlated input quantities is given by joint Gaussian distribution one can use the Cholesky procedure, which decom-poses the uncertainty matrix Ux for the values of the input quantities such that Ux = RTUx = RTR; where R is a matrix for which all elements below its diagonal are zero. The product of RT with values sampled from the PDF in standard form yields the sample values needed. For an illustration of the formalism, we discuss briefly the case of two correlated input quantities:
One obtains:
where we set r(x1,x2)=sinα and the ξstd,i,m are independent samples from a Gaussian in standard form, i.e. σi =1 and EXi = 0.If the correlation coefficient is given and if the shape of the PDF for the quantity that is the logical cause of correlation is known, then one can introduce it as auxiliary variable and consider Xi as an appropriately weighted sum or difference of its uncorrelated part Xi,0 and the quantity Xc, that causes the correlation.
( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛=
22
2211
221112
xuxux,xrxu
xux,xrxuxuXU
( )( ) ( ) ⎟⎟
⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛+⎟⎟
⎠
⎞⎜⎜⎝
⎛=⎟⎟
⎠
⎞⎜⎜⎝
⎛
m,
m,,
m,
m,
xuxuxu
xx
2std,
1std
22
1
2
1
2
1
cos sin0
ξξ
ααξξ
13
13The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
( )flags,,,E,k,t,p Ωrs =T
A history is manipulated to yield useful information; this is the reason for p in s.
Monte Carlo Method Sorting & Statistics (3/3) (7/7)
1 2
Step by step simulation: Generation of a history (2/2).
Concurrent example: every• started deuteron is to produce a neutron• neutron is to reach the sample and• detector is to be reached by scattered neutrons.
Enforcing is “paid” for bymultiplying the probability with
Generally, in a virtual experiment one needs methods to reduce variance.
SampleSampleGas target
xx
zz
D(d,n) X(n,n')X(n,n')
DetectorDetector
ϑn,n'ϑn,n'
A site r X(n,n’) for a scattering event , the nucleus and the scattering reaction be given.Select a point on the surface of the detector r Detector that is far away.
reduce ⇒ redistribute
X
Caveat: Method is in principle simple, but there are pitfalls.
( )2
Detector Sample
Detector
Reaction
ReactionDetector Sample 4cos
cos
i
ii r
A,Epi
→→ ∂
∂=
πϑσϑσ
ϑ
Stratified sampling (as one example of Variance reduction techniques)
The path length between two reactions of a neutron in matter, (viewgraph 6) leads to an exponential expression of the form:
λ is termed mean free path. We consider the path length of a neutron normally incident on slab with a thickness lmax. The cumulative distribution function can be written in closed form:
The inverse exists and we obtain a representative sample of the path length: ξ = −λ loge(1-ρ). However, for any ρ > 1−exp(−λ−1 lmax) the neutron would pass through the slab with any interaction. To avoid this waste of CPU time, one enforces an interaction in the slab. To that purpose we re-normalise the PDF:
and give a sample the “statistical weight” w= 1−exp(−λ−1 lmax). This can be generalised to selecting different path-length intervals with differentweights, large weights there, where the function varies strongly.
( ) ( ) ( );,,E-g X mattern whereexp ntot11 Σ== −− λξλλξ
( ) ( ) ( ).--GX ξλξξλλξξ
1
0
1 exp1 dexp −− −=′′= ∫
( ) ( ) ( )
if 0
if expexp1
1 11
⎪⎩
⎪⎨
⎧
>
≤−=
−−
max
maxmaxX
l
l-l-g
ξ
ξξλλλξ
14
14The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
The discussion in recent years and the forthcoming additional guidance by theJoint Committee Guides for Metrology (JCGM) and the definition of uncertainty
allow now to the concept of the GUM quite briefly:
Evaluation of Uncertainty GUM concept (1/3) (1/x)
1 2 3
Uncertainty of measurement (VIM Definition)parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand.
See WS-35 by M. G. Cox, P. M. Harris and I. M. Smith: Supplements to the GUM.
additional guidance definition of uncertainty
Modelof the measurement:
PDFs (probability density function)
for the input quantities:
reflects the knowledgeabout the interrelationof the input quantities Xi with the output quantity Y
( )NX,...,XfY 1= free form
( )N,...,f ξξη 1=uniquely
but fromgiven valuesfor the Xi
reflects our degree of belief in reasonably possible values, based on given information.
Model and PDF
reflects the knowledgeabout the interrelationof the input quantities Xi with the output quantity Y
physical,chemical,biological,…
Result is determined, once model and PDFs are given, it is calculated, not estimated!
GUM concept (brief):The knowledge given on possible values ξi of the input quantities Xi is expressed by PDFs and the knowledge given on their interrelation with the values η of the output quantity (measurand) Y is expressed by the model function f. The values η can be expressed via the model as a function of the values ξi. The form of the model is not prescribed. However, the model must reflect all influencing quantities, including corrections. Otherwise one could loose traceability! A closed theory for modelling is not known.The PME and the Bayes theorem are used to derive a PDF from data. It is important to realise, that a well defined measurand + has one value, but our knowledge is in principal incomplete, whence we can,if we are honest, only express the degree of believe in possible values based on given data* and never state one and only one value.+It is hard to define a measurand well. Consider as simple example the length ofan infinitesimal thin, but perfectly straight, rigid and incompressible wire at a temperature zero Kelvin in absolutely free space!
*Data pl. of datum (Latin: given, cited from Webster): something given or admitted especially as a basis for reasoning or inference.
15
15The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
M
TMTM
TAirTAir
mIND
F=MgF=Mg
F→LF→L
ρMρM
ρAirρAir
∆L∆L
δLδL
........bouancybouancy couplingcoupling deviationsdeviations(transfer)(transfer)
0 0
5 5
1010
1515kgkg
( )MINReading,MAXReading,0 mml −=
Evaluation of Uncertainty GUM concept (2/3) (2/x)
1 2 3
“Model” demonstrates:even simple models, that neglectmany influences, need furthersimplification and are not linear.
Just for demonstration!
mgBF =
⎟⎟⎠
⎞⎜⎜⎝
⎛−=
m
Bρρa1
INDINDScaleIND δ∆ mmmm ++=Approximation:CF→L = l0 α (F/2)
& F ≅ mIND E(gB)
LFFCm →=Scale
Model: example
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛′′+= ∫
+
→ FT,FlCFF
FLF d1
0
0
0 α
It is good practice to break down complicated models into sub-models.
( ) ( ) E2
δ∆ 1
IND0INDINDIND
−
⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛−−=
gBmgBlmmmm α
Measurand is m:
δ∆ INDIND mgBCmmm LFS −−= →
Calibration:mIND & E(gB) are const., need PDFs for others!
buoyancy
Clearly, the above “model” is contrived, since one would turn to an alternative measurement principle and measurement method, if one were to aim at such small uncertainties that all of the discussed influences become significant.But, it is hopefully didactical, as all discussed effects can be understood in basic physical terms.Stated without proof, Monte Carlo integration could even tackle
where F is force and T is temperature.
Admittedly, it would be hard work. However, a detailed study on the above used approximation could be carried out using Monte Carlo. Such a study could result in in a parameterisation of a PDF for CF→L
as a function of F, or , if needed of F and T.There may be many cases in practice, where it could be necessary, to use Monte Carlo techniques for studying whether or not made approximations in modelling are justified.
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛′′+= ∫
+
→ FT,FlCFF
FLF d1
0
0
0 α
16
16The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Evaluation of Uncertainty GUM concept (3/3) (3/x)
Consider a model and PDFs for the input quantities Xi are given:(Information of TYPE A and TYPE B leads via PME and Bayes theorem to PDFs.)
Combining PDFs
1 2 3
How can one determinepossible values of Y, η ?
( ) ξξξ dE ∫∞
∞−
==iXii gXx ( ) ( )( ) ξξξ dVar 22 ∫
∞
∞−
−== xgXxuiXii( )321 X,X,XfY =
ξ1
ξ2
ξ3
ηm = f (ξ1,m, ξ2,m, ξ3,m )
ξ1,m ξ2,m ξ3,m
Representative: e.g. analogue sampling(see viewgraph 10)Fo
rwar
d ca
lcul
atio
n
* Reminder: Knowledge about values is expressed by PDF
*For determining the coverage interval we need gY(η).
By representative combinationsof all possible values of the inputquantities, according to the model !
Markov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫∞
∞−X
If the dice have an increasing number of faces, then the resulting PDF for the sum will tend to a triangular PDF. If one uses dices with different numbers of faces one obtains a trapezoidal PDF.
00 11r=(2π)-1r=(2π)-1
+
)(f 21 ξξη +=
Possible values of Y and
do depend on + – . or /, but the number of combinations does not!
Probabilities for values
2211 33 44 55 66 2211 33 44 55 66
22 33 44 55 66
2211 33 44 55 66
77 88 99 1010 1111 1212
361
362
363
364
365
366
365
364
363
362
361
2211 33 44 55 66 2211 33 44 55 66
22 33 44 55 66
2211 33 44 55 66
77 88 99 1010 1111 1212
361
362
363
364
365
366
365
364
363
362
361
361
362
363
364
365
366
365
364
363
362
361
361
362
363
364
365
366
365
364
363
362
361
Consider two fair dice with six faces each
Assume now a continuous fair dice:all values between 0 and 1 are equally probable, this leads to a uniform PDF and a triangular PDF for Y.
[ ]10,i ∈ξ
Illustration of combining possiblevalues from two PDFs, here discrete.Try it!
17
L
p
BB b
h
QQ BB b
Construction causes a raise Construction causes a raise of water level (Bernoulli).of water level (Bernoulli).
23
23
32 hbCCgQ Dv⎟⎠⎞
⎜⎝⎛=National Engineering Laboratory, UK
Flow rate in a cooling water channel
23
0030100601 ⎟⎠⎞
⎜⎝⎛ −⎟⎠⎞
⎜⎝⎛ −=
hL.
bL.CD01
332
322
22
=+−⎟⎟⎠
⎞⎜⎜⎝
⎛+⎟
⎟⎠
⎞⎜⎜⎝
⎛vv CC
phh
Bb
? =∂∂
BQ
17The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Evaluation of Uncertainty Markov Formula & Standard Procedure (4/x)
1 2 3
-2 0 20
0.2
0.4
g X i( ξ i
)
-1 0 1ξ i / u( xi)
-2 0 20
0.5
1
GX
i (ξ i
)
0 100 200 300Bin k
0
5
10
g Y (k
)
0
103
← M=1000← M=1000 M=1000000 →M=1000000 →
Markov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫∞
∞−XMarkov formula: NNNY ξξξfξgg ...d))d,...,((),...,()( 111 ξηδξη −= ∫
∞
∞−X
ξξξX d )()( fgy ∫∞
∞−
=
( ) ( )[ ] ξξξX d)( 22 -yfgyu ∫∞
∞−
=
Integration over η:
( ) ( )∑=
−+=N
iiii xcf
1 ξη x
In general y ≠ f (x),but for linear model:
since E (ξi− xi) = 0
vector
Linear model in Markov formula ⇒ u2(y) = cT UX c, i.e. GUM standard in matrix form
cT=(c1,...,cN)uX,i,j= u(xi,xj)
Frequency distribution⇒ PDF for measurand
η =ηoffset + ck→η k
An example of a problem, where the numerical calculation of at least onesensitivity coefficient is fairly difficult.The example was originally provided by the National EngineeringLaboratory, UK.An obstruction in a cooling water channel (supply for a power station) leads to a measurable raise of the water level. x x
18
18The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Evaluation of Uncertainty Advantages of MCI ? (5/x)
1 2 3
Monte Carlo integration:
Key comparisons
Model is non linear
PDF for Y is not a Gaussian
More than one output quantity
Sensitivity coefficients.
Real world is not alwaysGaussian
( ) 950dmax
min
.gy
yY =∫ ηη
Find min ymax− ymin
Dominating uniform or t-PDF,Asymmetric PDF.
E.g. compute median or pooled data.
Straightforward, but problem to represent U.
1nwi =
( )nY,.....,Y,YfZ 21≡ ∑ ⋅=n
iii ywz
∑ ⋅=n
ijiij w ,ηζ
( )
( )∑′
′−
−
⋅= n
ii
ii
yu
yuw2
2
Least squares
( ) odd if 121 NNi*
sort +⋅=
121 2
1 sortsort +⋅=⋅= +− NiNi
Median: sort ζ i and take
Detailed variance analysis and search formodel simplification
k=U/u(y)?
⇒ U− and U+
Supplement:
Monte Carlo sampling allows to compute many interesting feature, such as the variance or the relative weights of the participants: Count the number mPi of samples where Participant i is the median or (even number of participants) contributes to the median: mPi /M is thentermed weight; M= total number of samples.Note: The median for correlated input quantities has peculiar features.
Also being discussed in literature is the median
( )
⎪⎭
⎪⎬⎫
⎪⎩
⎪⎨⎧
+⋅=⋅=⎟⎠⎞⎜
⎝⎛ +⋅
+⋅==
+−+
− even if 121 and 2
1 ; 21
odd if 121 ;
nnini
nni
sortsortii
*sort*i
sortisortηη
ηζ
The values of the ηi obtained in the mth given historyare increasingly ordered, e.g.
η2 ≤ η3 ≤ η1 ≤ η7 ≤ η4 ≤ η6 ≤ η5 n = 7 (odd)
The mth value for ζ is the given by:
η2 ≤ η3 ≤ η1 ≤ η7 ≤ η4 ≤ η6 ≤ η5 ≤ η8 n = 8 (even)
19
19The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Simulation („virtual experiment“) Model (1/4)
1 2 3 4
Monitor
Monitor
Cyclotron
Beam-lineTarget
Sample
D(d,n)D
etec
tors
DetectorDetermination of differential neutron cross section
via a Time of flight (TOF) experiment
“realistic view”
(n, n’)
Target
SKIP 20
Collimator
Details on the simulation of the TOF-experimentA cyclotron is used to accelerate deuterium ions (D+) . The extracted beam is directed to a deuterium gas target, where neutrons are produced. The neutrons are emitted in all directions, some will hit a sample, say consisting of polyethylene or other materials of interest. The neutrons interact with that material and are scattered. Some of these neutrons are detected by detectors under different angles. A collimator assures that the detectors “see” only the sample and not the target. A calibratedmonitor, shielded such as to “see” only the target, is used to link the measurement to neutron fluence. The cyclotron, together with beam line and target, can be turned around the sample, such that the set of five detectors allows scattering angles from about 120 to 1650 to be obtained.The simulation starts by selecting a deuteron with a given time (of “birth”), energy and location. These data are sampled from PDFs that are derived from emittance measurements. The deuterons are transported to the target, they loose some energy in the gas and are forced to generate a neutron in an spatial angle that encompasses the sample. The neutrons are scattered in the sample and then transported to the detectors.The important variable is the time (of flight) that elapses between thebirth of a deuteron and the detection of a neutron in a detector.(see also bottom of page 22)
20
20The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
xx
z
y
ϑiϑi
MonitorDetector i
Detector j
ϑjϑj
SampleSample
Gas target
Simulation („virtual experiment“) Model (2/4)
1 2 3 4
d passes through fixed plane
td,0 , Ed,0 , rd,0, cd,0
d is slowed down in foil andgas prior to possible reaction
td,1 , Ed,1 , rd,1, cd,1
x
d reacts: D(d,n)
tn,1 , En,1 , rn,1, cn,1
Enforce n to react in monitorEnforce n to interact in sample
Enforce n‘ to interact in detector 1
Enforce n‘ to interact in detector n...............Iterate
Compute TOF spectra:
Total, Mul 1,2,..mDiscriminate reaction typesand compute spectators.
Gas target
Determination of differential neutron cross section via a Time of flight (TOF) experiment
steps for computation(overview only)
Sort event in spectra
Details on the simulation of the TOF-experiment(continued from page 19)Details on treating the gas targetThe entrance window of the gas target (needed since the beam line is evacuated) is very thin and bent out toward the vacuum.The deuterons are slowed down in that entrance window and there is not only a loss in energy, but also energy and angular straggling.Very few deuterons react in the gas target. This allows to approximate the loci of interaction by an rectangular PDF and to use a simple energy loss model.The reaction D(d,n) is not isotropic. The code samples a location on a circle (normal coincident with beam line) that is circumscribed about the sample. This fixes the direction of the newly “born” neutron and the spatial angle. Since the sample is a cylinder, not all neutrons hit it. They need to be rejected and this leads to a correction of the spatial angle that is used for computing the statistical weight of the neutron.Details on treating the sampleNeutrons, that hit the sample are forced to react. Special care is needed for neutrons that intersect the top or the bottom of the sample cylinder. Furthermore, some samples are liquids and then one needs to also account for scattering in the flask (closed thin metallic cylinder).
21
21The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Simulation („virtual experiment“) Inference (3/4)
1 2 3 4
( ) ΩΩσΩΩσ ′⋅′⋅′′⋅′′< ∫∫rrrr
dd)( = >)( prior r,Simulationexp,Simulationexp,r E,E,Eg,E
K(ϑi) is obtained from fittingthe simulated spectra to theexperimental ones.
Simplified: The sum of themeasured events in the peakspertaining to the reaction understudy is normalised to 1.
The sum of the simulatedevents in the peaks pertaining tothe same reaction is normalisedto 1.
( ) Literatur""reactionneureaction )( )( iii ,EK,E ϑσϑϑσ ⋅=
Determination of differential neutron cross section via a Time of flight (TOF) experimentTOF
Cou
nts
per c
hann
el
C(n,n’) H(n,n’)
Edge is used to validatecyclotron position! (“intrinsic-calibration”)
GOTO ENDE
Details on the simulation of the TOF-experiment(continued from page 20, Details on treating the sample)How an intersection site is generated is explained on page 13 (bottom).If the interaction site is selected, a nucleus is to be selected as reaction partner.If the nucleus is selected, a possible reaction is to be selected, for instance elastic scattering or inelastic scattering; the latter can have different Q-values (for different excitation levels).Details on treating the detectorsFrom each interaction site (nucleus and reaction are selected) the code computes then the probability to reach a detector for this neutron. This is done for up to fifty detectors under different angles. The neutrons are sorted in tallies according to their time of flight.Then multiple scattering is enforced in the sample and the probability of detection by the detectors is again computed, This iteration is ended by various cut-off conditions.The needed PDFs are generated from cross section data found in evaluated neutron data file (ENDF) and stopping power tables. The geo-metry data are taken from the construction sheet, the uncertainties are not treated, but there influence was examined.
22
22The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Simulation („virtual experiment“) Validation and uncertainty (4/4)
1 2 3 4Determination of differential neutron cross section via a Time of flight (TOF) experiment
Literature, start valuesFinal values
Reproduction of measurements validates code Sources of uncertainty in a simulation:algorithmstochasticestimatorsapproximations.....
parameters:geometrymaterial compositionsstopping powerscross sections
......
Brute force,correlated samplingand perturbationtechniques
Advantages: Optimisation of the experiment, “spectators”calculation of not measurable coefficients,...
Costs: Computing times, development, but: well-maintained and quite flexible code packages exist, e.g. MCNP for radiation transport.
Edge is used tovalidatecyclotronposition!
Annotations to spectators for optimisationThe accumulated tallies, i.e. the time of flight spectra for each detector, are then the basis for the evaluation as briefly outline on viewgraph 21.The main advantage of this simulation method is, that one can compute many so-called spectators that provide valuable information, which couldnot be obtained from the real experiment.Example are:
extra tallies for total spectra 1st scattering for each detector and
…………… for each reactionmth scattering.
tallies that discriminate against geometry, regions in targetregions in sample.
tallies for the statistical weight distribution in time of flightwindows to assess uncertainties
An additional power of such a simulation is the possibility to use it to optimise the measurement, for instance by searching for the optimal detector positions, optimal target-sample distance and sample form.
23
23The use of Monte Carlo techniques for metrology
- evaluation of uncertainties and simulation of experiments -
Summary and Conclusions
51 2 3 4
Thanks for your attention.
Conclusion:Monte Carlo techniquesadd value to metrology sincethey open up new ways formodelling and “exact” solutions.
0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
Measurement 0 0
5 5
1010
1515kgkg
MeasurementMM
M MM
Evaluation of uncertainty 0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
0 0
5 5
1010
1515kgkg
Measurement 0 0
5 5
1010
1515kgkg
MeasurementMM
M MM
Evaluation of uncertainty
SampleSampleGas target
xx
zz
D(d,n) X(n,n')X(n,n')
DetectorDetector
ϑn,n'ϑn,n'
Simulation („virtual experiment“)
Monte Carlo techniques are ideal tools forcomputing the PDF for the measurand(Markov formula: “propagation” of PDFs)
simulating complicated experiment(Necessary for evaluation)
in depth uncertainty analysismodel simplifications
Approximation: CF→L = l0 α (F/2)and F≅ mIND gB
Final remarkThe method can also be used to optimise the model for the evaluation of uncertainty, An indication was given on bottom of page 15. This might be even of more interest with respect to model for key comparisons. Also, if the PDFs for the participants’ measurand are not Gaussian, or if there is correlation, Monte Carlo techniques could help to find the best model and solutions.Even more important, the method does not need to rely on statistics theorem that are often only valid for certain PDFs, mostly Gaussians, and very large numbers. One can simply reiterate the entire calculation in order to get numerical representations. For instance: What is the correct uncertainty if you have correlated few repeated measurement, drawn from an uniform PDF? Simply draw a couple of hundred million time such samples and sort them (takes only a few hours on a modern PC). The extension to more than one output quantity is trivial.Above all these advantage, the method is also intellectually pleasing, as it is simple and implements Bayesian inference in direct way with given priors (see Introduction).Nevertheless, do not use Monte Carlo, if you can get analytic solutions.But, use always (partial) analytic solutions to validate your code and assess the uncertainties contributed by the algorithms used. [email protected]