optimization by marker removal for δf particle simulations
DESCRIPTION
A marker removal optimization technique is developed for δf particle simulations. The technique uses the linear eigenmode structure in the equilibrium constant-of-motion space to construct an importance function, then removes some markers based on the importance function and adjusts the weights of the leftover markers to optimize the marker distribution function, so as to save markers and computing time. The technique can be directly applied to single-mode linear simulations. For multi-mode or nonlinear simulations, the technique can still be directly applied if there is one most unstable mode that dominates the simulation and δf does not change too much in the nonlinear stage, otherwise special care is needed, which is discussed in detail in this paper. The technique’s effectiveness, e.g., marker saving factor, depends on how localized δf is. The technique can be used for a phase space of arbitrary dimension, as long as the constants of motion in equilibrium can be found. In this paper, the technique is tested in a 2D bump-on-tail simulation and a 5D gyrokinetic toroidal Alfvén eigenmode (TAE) simulation and saves markers by factors of 4 and 19, respectively. The technique is not limited to particle-in-cell (PIC) simulations but could be applied to other approaches of marker particle simulations such as particle-in-wavelet (PIW) and grid-free treecode simulations. This set of slides has been presented at the 23rd International Conference on Numerical Simulation of Plasmas (2013). The corresponding paper has been published on Computer Physics Communications 185, 96–105 (2014): http://wdeng.info/publications/optimization-by-marker-removal-for-%CE%B4f-particle-simulations/TRANSCRIPT
Optimization by marker removal for δf particlesimulations
Wenjun Deng and Guo-Yong FuPrinceton Plasma Physics Laboratory
Supported by U.S. DOE SciDAC Center for Nonlinear Simulationof Energetic Particles in Burning Plasmas (CSEP)
MotivationComputational particles called markerparticles (or markers) are popularly used forMonte Carlo sampling of plasma distributionfunction in plasma simulations, e.g.,particle-in-cell (PIC) simulations.
In δf marker simulations, f = f0 + δf ,markers are for resolving δf structure inphase space.
Since δf is unknown beforehand, typically ahuge number of markers are loadedeverywhere in phase space.
For some modes, e.g., energetic particleexcited Alfven eigenmodes, δf ≈ 0 in a largeportion of the phase space. Maintaining alarge number of markers there is a waste ofcomputing time.
Reducing number of markers in the δf ≈ 0regions saves computing time.
An example of localization ofδf structure in TAE
simulation by M3D-K[Lang et al., PoP 2010]
1/19
Some candidate techniques
Marker increase: initially load insufficient markers,
then split/reload markers to increase resolution
where δf structure is localized;
I splitting can easily introduce non-conservationproblem if not treated carefully [Lapenta, JCP2002], bringing in systematic error
I children markers split from one parent markerare correlated, thus can’t reduce noise untilafter decorrelation time (could be long)
I reloading (phase space remapping) [B. Wang,SIAM JSC 2011 & 2012] is a better way inthis category
Marker reduction: initially load sufficient markers,
then merge/remove markers in δf ≈ 0 regions.
I finding nearby particles to merge is not veryeasy, especially in high dimensional phasespace
In this work, a removal technique is developed.
δf
split
merge
remove
initial
optimized
initial
optimized
initial
optimized
2/19
Some candidate techniques
Marker increase: initially load insufficient markers,
then split/reload markers to increase resolution
where δf structure is localized;
I splitting can easily introduce non-conservationproblem if not treated carefully [Lapenta, JCP2002], bringing in systematic error
I children markers split from one parent markerare correlated, thus can’t reduce noise untilafter decorrelation time (could be long)
I reloading (phase space remapping) [B. Wang,SIAM JSC 2011 & 2012] is a better way inthis category
Marker reduction: initially load sufficient markers,
then merge/remove markers in δf ≈ 0 regions.
I finding nearby particles to merge is not veryeasy, especially in high dimensional phasespace
In this work, a removal technique is developed.
δf
split
merge
remove
initial
optimized
initial
optimized
initial
optimized
2/19
Some candidate techniques
Marker increase: initially load insufficient markers,
then split/reload markers to increase resolution
where δf structure is localized;
I splitting can easily introduce non-conservationproblem if not treated carefully [Lapenta, JCP2002], bringing in systematic error
I children markers split from one parent markerare correlated, thus can’t reduce noise untilafter decorrelation time (could be long)
I reloading (phase space remapping) [B. Wang,SIAM JSC 2011 & 2012] is a better way inthis category
Marker reduction: initially load sufficient markers,
then merge/remove markers in δf ≈ 0 regions.
I finding nearby particles to merge is not veryeasy, especially in high dimensional phasespace
In this work, a removal technique is developed.
δf
split
merge
remove
initial
optimized
initial
optimized
initial
optimized
2/19
Some candidate techniques
Marker increase: initially load insufficient markers,
then split/reload markers to increase resolution
where δf structure is localized;
I splitting can easily introduce non-conservationproblem if not treated carefully [Lapenta, JCP2002], bringing in systematic error
I children markers split from one parent markerare correlated, thus can’t reduce noise untilafter decorrelation time (could be long)
I reloading (phase space remapping) [B. Wang,SIAM JSC 2011 & 2012] is a better way inthis category
Marker reduction: initially load sufficient markers,
then merge/remove markers in δf ≈ 0 regions.
I finding nearby particles to merge is not veryeasy, especially in high dimensional phasespace
In this work, a removal technique is developed.
δf
split
merge
remove
initial
optimized
initial
optimized
initial
optimized
2/19
Outline
1 Start with a simple plasma problem: 2D bump-on-tail simulation
2 Optimization by marker removal in 2D phase space
3 Generalize to 5D gyrokinetic phase space
4 Summary and possible extensions
3/19
Outline
1 Start with a simple plasma problem: 2D bump-on-tail simulation
2 Optimization by marker removal in 2D phase space
3 Generalize to 5D gyrokinetic phase space
4 Summary and possible extensions
2D electrostatic Vlasov-Poisson PIC simulation modelVlasov-Poisson equations (α indicating species):
dfα(x, v, t)
dt= ∂tfα(x, v, t) + v∂xfα(x, v, t) +
ZαE(x, t)
mα∂vfα(x, v, t) = 0
∂xE(x, t) =∑α
Zα
∫ ∞−∞
fα(x, v, t)dv
Normalization: time: 1/ωpe; length: λDe; velocity: vth,e; mass: me; charge: e;
temperature: Te; density: ne; E-field: e/(TeλDe)
δf model (f = f0 + δf):
d
dtδfα = − d
dtf0α = − Zα
mαE∂vf0α
∂xE(x, t) =∑α
Zα
∫ ∞−∞
δfα(x, v, t)dv '∑α
Zα∑jα
wjαS(x− xjα)
Marker motion equations for PIC simulation (p = f/g, w = δf/g):
dxjαdt
= vjαdvjαdt
=Zαmα
E(xjα)
dpjαdt
= 0dwjα
dt= −(pjα − wjα)
Zαmα
E(xjα)∂vf0αf0α
S(): shape function; g: marker distribution; jα: marker index
4/19
Analytic dispersion relation of bump-on-tail instability
Bump-on-tail distribution: sum of two (shifted) Maxwellian:
f0e(v) =1√2π
(1− nb) exp
(−v
2
2
)︸ ︷︷ ︸background Maxwellian
+nbvth,b
exp
[− (v − v0b)2
2v2th,b
]︸ ︷︷ ︸
beam shifted Maxwellian
Analytic dispersion relation for a plasma composed of (shifted)Maxwellian species:
1 +1
k2
∑α
nαZ2α
Tα[1 + ζαZ(ζα)] = 0
ζα =ωk − v0α√
2 vth,αZ() : plasma dispersion function
Mode structure in phase space:
δfα(x, v) =nαZαTα
v − v0α√2π vth,α
exp
[− (v − v0α)2
2v2th,α
]i
ω − kvE0e
ikx−iωt + c.c.
5/19
Linear benchmark of PIC simulation vs analytic solution
nb = 0.1, v0b = 5, v2th,b = 0.1, k = 0.401, γ/ωr = 2%, vres = 3.6593
Semi-analytic solution:ωr = 1.4674γ = 0.029120
PIC simulation:ωr ≈ 1.47γ ≈ 0.0287
0 π/k 2π/kx
8642
024
68
v
(a)
0 π/k 2π/kx
(b)
δf structure from semi-analytic calculation and PIC simulation
6/19
Outline
1 Start with a simple plasma problem: 2D bump-on-tail simulation
2 Optimization by marker removal in 2D phase space
3 Generalize to 5D gyrokinetic phase space
4 Summary and possible extensions
Marker removal algorithmMarker removal is performed in the early linear stage, when the lineareigenmode just forms.
Procedure: for each marker jα:1 quantify its importance Ijα in the range of [0, 1], with 0 indicating least
important, 1 indicating most important;2 generate a uniform [0, 1] random number, called dice;3 if dice > Ijα , remove this marker; otherwise, keep this marker and scale
up its weights: pjα → 1Ijα
pjα , wjα → 1Ijα
wjα .
(This way, only statistical error is introduced to the distribution function, as well as
mass, momentum and energy)
Calculation of importance Ijα :I Unnormalized importance function:
iα(v) =
∫|δfα(x, v)|dx∫gα(x, v)dx
≈∑jα|wjα |S(vjα − v)∑jαS(vjα − v)
g: marker distribution; S(): shape functionI Normalized importance function:
Iα(v) =iα(v)
max{iα(v)}I Ijα is calculated from normalized importance function: Ijα = Iα(vjα)
7/19
Marker removal algorithmMarker removal is performed in the early linear stage, when the lineareigenmode just forms.
Procedure: for each marker jα:1 quantify its importance Ijα in the range of [0, 1], with 0 indicating least
important, 1 indicating most important;2 generate a uniform [0, 1] random number, called dice;3 if dice > Ijα , remove this marker; otherwise, keep this marker and scale
up its weights: pjα → 1Ijα
pjα , wjα → 1Ijα
wjα .
(This way, only statistical error is introduced to the distribution function, as well as
mass, momentum and energy)
Calculation of importance Ijα :I Unnormalized importance function:
iα(v) =
∫|δfα(x, v)|dx∫gα(x, v)dx
≈∑jα|wjα |S(vjα − v)∑jαS(vjα − v)
g: marker distribution; S(): shape functionI Normalized importance function:
Iα(v) =iα(v)
max{iα(v)}I Ijα is calculated from normalized importance function: Ijα = Iα(vjα)
7/19
Marker removal in bump-on-tail simulation
10−1010−910−810−710−610−510−410−310−210−1
0 100 200 300 400 500 600 700 800 9001000
∫ E2dx
t
Remove markersonce linear eigenmode is formed
0 π/k 2π/kx
864202468
v
(a)1.621.801.982.162.342.522.702.883.063.24
×106
0 π/k 2π/kx
864202468
v
(b)0.00.30.60.91.21.51.82.12.42.7
×106
∫g dx
864202468
v(c)
0 π/k 2π/kx
864202468
v
(d)
Marker distribution before and after removal and compare with δf8/19
How to evaluate an optimization technique?
Optimization goal: reduce numericalerror for the same marker number, orretain the same numerical error withless markers.
Need to measure numerical error due tofinite marker number.
Because marker simulation is essentiallya Monte Carlo simulation, thenumerical error is mainly statisticalrather than systematic, a traditionalsingle-scan convergence test does notmeasure numerical error accurately.
I Convergence trend may be notclear
I The statistical error ∼ 1/√Nm is
not revealedI Inconsistency because error is
mainly statistical (random)
2.852.8552.86
2.8652.87
2.8752.88
2.8852.89
2.8952.9
1 10 100 1000
γ/1
0−2
# of markers Nm (million)
(c)
2.75
2.8
2.85
2.9
2.95
3
3.05
γ/10
−2
(b)
2.822.842.862.882.92.922.942.962.98
γ/10
−2
(a)
Traditional single-scan convergencetests of bump-on-tail growth rategive different results when markersare initialized by different randomnumber sequences
9/19
How to evaluate an optimization technique?
Optimization goal: reduce numericalerror for the same marker number, orretain the same numerical error withless markers.
Need to measure numerical error due tofinite marker number.
Because marker simulation is essentiallya Monte Carlo simulation, thenumerical error is mainly statisticalrather than systematic, a traditionalsingle-scan convergence test does notmeasure numerical error accurately.
I Convergence trend may be notclear
I The statistical error ∼ 1/√Nm is
not revealedI Inconsistency because error is
mainly statistical (random)
2.852.8552.86
2.8652.87
2.8752.88
2.8852.89
2.8952.9
1 10 100 1000
γ/10
−2
# of markers Nm (million)
(c)
2.75
2.8
2.85
2.9
2.95
3
3.05
γ/10
−2
(b)
2.822.842.862.882.9
2.922.942.962.98
γ/10
−2
(a)
Traditional single-scan convergencetests of bump-on-tail growth rategive different results when markersare initialized by different randomnumber sequences 9/19
A more rigorous convergence test is needed
A more rigorous convergence test:I performs simulations multiple times
(with markers initialized by differentrandom number sequences) as asample for every marker number;
I uses sample statistics to estimatenumerical error;
I can verify that error due to finitemarker number is mainly statistical;
I can reveal the inverse-square-rootrelation between statistical error andmarker number.
Optimization goals:I make data points in Fig (a) stay at
the same level, so that theoptimization doesn’t introduceapparent systematic error;
I lower data points in Fig (b), so thatstatistical error is reduced.
1
10
1 10 100 1000
stat
isti
cal
un
cert
ain
ty(%
)ofγ
# of markers Nm (million)
(b)
sample std for stat.uncertainty estimate
fitting N−1/2m
2.82
2.84
2.86
2.88
2.9
2.92
2.94
2.96
2.98
γ/1
0−2
(a)
sample mean forexpectation estimate
A more rigorous convergence testof bump-on-tail growth rate
10/19
A more rigorous convergence test is needed
A more rigorous convergence test:I performs simulations multiple times
(with markers initialized by differentrandom number sequences) as asample for every marker number;
I uses sample statistics to estimatenumerical error;
I can verify that error due to finitemarker number is mainly statistical;
I can reveal the inverse-square-rootrelation between statistical error andmarker number.
Optimization goals:I make data points in Fig (a) stay at
the same level, so that theoptimization doesn’t introduceapparent systematic error;
I lower data points in Fig (b), so thatstatistical error is reduced.
1
10
1 10 100 1000
stat
isti
cal
un
cert
ainty
(%)
ofγ
# of markers Nm (million)
(b)
sample std for stat.uncertainty estimate
fitting N−1/2m
2.82
2.84
2.86
2.88
2.9
2.92
2.94
2.96
2.98
γ/10−
2
(a)
sample mean forexpectation estimate
A more rigorous convergence testof bump-on-tail growth rate
10/19
A more rigorous convergence test is needed
A more rigorous convergence test:I performs simulations multiple times
(with markers initialized by differentrandom number sequences) as asample for every marker number;
I uses sample statistics to estimatenumerical error;
I can verify that error due to finitemarker number is mainly statistical;
I can reveal the inverse-square-rootrelation between statistical error andmarker number.
Optimization goals:I make data points in Fig (a) stay at
the same level, so that theoptimization doesn’t introduceapparent systematic error;
I lower data points in Fig (b), so thatstatistical error is reduced.
1
10
1 10 100 1000
stat
isti
cal
un
cert
ainty
(%)
ofγ
# of markers Nm (million)
(b)
sample std for stat.uncertainty estimate
fitting N−1/2m
2.82
2.84
2.86
2.88
2.9
2.92
2.94
2.96
2.98
γ/10−
2
(a)
sample mean forexpectation estimate
A more rigorous convergence testof bump-on-tail growth rate
10/19
Marker removal effect on growth rate
1
10
10 100 1000
statistical
uncertainty
(%)
ofγ
# of markers Nm (million)
(b)
regularoptimized
N−1/2m , regular
N−1/2m , optimized
2.82
2.84
2.86
2.88
2.9
2.92
2.94
2.96
2.98
γ/10
−2
(a)
regularoptimized
Marker removalI does not introduce
apparent systematicerror to growth rate;
I saves markers by afactor of 4.
11/19
Marker removal effect on saturation level
1
10
100
1 10 100 1000
stat.uncertainty
(%)
ofsat.
level
of∫ E
2dx
# of markers Nm (million)
(b)
regularoptimized
N−1/2m , regular
N−1/2m , optimized
0.9
0.95
1
1.05
1.1
1.15
1.2
1.25
saturation
level
of∫ E
2dx/10
−2
(a)
regularoptimized
Marker removalI does not introduce
apparent systematicerror to saturationlevel;
I saves markers by afactor of 4.
12/19
Outline
1 Start with a simple plasma problem: 2D bump-on-tail simulation
2 Optimization by marker removal in 2D phase space
3 Generalize to 5D gyrokinetic phase space
4 Summary and possible extensions
Importance function in 5D gyrokinetic phase space
In phase space, some coordinates are constants of motion (denoted as zc), theothers are non-constants (denoted as zn). In 5D gyrokinetic phase space,zc = (E , µ, Pζ), zn = (ζ, θorb).
Unnormalized importance function becomes:
iα(zc) =
∫|δfα(zc,zn)J |dzn∫gα(zc,zn)J dzn
≈∑jα|wjα |S(zc,jα − zc)∑jαS(zc,jα − zc)
g: marker distribution; S(): shape function
Normalized importance function:
Iα(zc) =iα(zc)
max{iα(zc)}
In fact, this expression of importance function is not limited to 5D gyrokineticphase space, but is general for a phase space of arbitrary geometry anddimension, as long as zc and zn are replaced accordingly.
Procedure for the removal technique in 5D phase space is the same as that in2D phase space.
M3D-K solves energetic particles in 5D gyrokinetic phase space by PICmethod. This marker removal technique has been implemented in M3D-K.
13/19
Importance function in 5D gyrokinetic phase space
In phase space, some coordinates are constants of motion (denoted as zc), theothers are non-constants (denoted as zn). In 5D gyrokinetic phase space,zc = (E , µ, Pζ), zn = (ζ, θorb).
Unnormalized importance function becomes:
iα(zc) =
∫|δfα(zc,zn)J |dzn∫gα(zc,zn)J dzn
≈∑jα|wjα |S(zc,jα − zc)∑jαS(zc,jα − zc)
g: marker distribution; S(): shape function
Normalized importance function:
Iα(zc) =iα(zc)
max{iα(zc)}
In fact, this expression of importance function is not limited to 5D gyrokineticphase space, but is general for a phase space of arbitrary geometry anddimension, as long as zc and zn are replaced accordingly.
Procedure for the removal technique in 5D phase space is the same as that in2D phase space.
M3D-K solves energetic particles in 5D gyrokinetic phase space by PICmethod. This marker removal technique has been implemented in M3D-K.
13/19
Marker removal in TAE simulation by M3D-K
10−1410−1310−1210−1110−1010−910−810−710−6
0 500 1000 1500 2000 2500
MHD
pert.energy
t/(vA/R0)
Remove markersonce linear eigenmode is formed
3rd peak
2.0 2.5 3.0 3.5 4.0R
1.0
0.5
0.0
0.5
1.0
Z
(a)2.0 2.5 3.0 3.5 4.0
R
(b)
n = 1 TAE linear and nonlinearmode structure
0.50 0.25 0.00Pζ
2
4
6
8
10
12
E
(a)
0.000.240.480.720.961.201.441.681.92
×106
0.50 0.25 0.00Pζ
2
4
6
8
10
12E
(b)
543618
01836547290108
0.50 0.25 0.00Pζ
2
4
6
8
10
12
E
(c)
0.00.61.21.82.43.03.64.24.85.4
×104
Marker distribution and δf before removal, and marker distribution after removal,in the µ = 0 slice of the zn-integrated phase space
14/19
Marker removal effect on TAE linear growth rate
1
10
0.01 0.1 1 10 100
statistical
uncertainty
(%)
ofγ
# of markers Nm (million)
regularoptimized
N−1/2m , regular
N−1/2m , optimized
6.4
6.45
6.5
6.55
6.6
6.65
6.7
6.75
6.8γ/10
−3
regularoptimized
Finite marker number doesnot introduce apparentsystematic error.
Statistical uncertainty goes∼ (# of markers)−
12 .
Marker removalI does not introduce
apparent systematicerror;
I saves markers by a factorof 19.
15/19
Marker removal effect on TAE nonlinear saturation level
10
100
0.01 0.1 1 10 100
statistical
uncertainty
(%)
ofsat.
lvl.(1st
peak)
# of markers Nm (million)
regularoptimized
N−1/2m , regular
N−1/2m , optimized
6.5
7
7.5
8
8.5saturation
level
(1st
peak)/10
−8
regularoptimized
Marker removalI does not introduce
apparent systematicerror;
I saves markers by a factorof 34.
16/19
Marker removal effect on TAE third nonlinear peak
10
100
0.01 0.1 1 10 100
statistical
uncertainty
(%)
of3rdpeakam
plitude
# of markers Nm (million)
regularoptimized
N−1/2m , regular
N−1/2m , optimized
2.6
2.8
3
3.2
3.43rdpeak
amplitude/10
−7
regularoptimized
Marker removalI does not introduce
apparent systematicerror;
I saves markers by a factorof 24.
17/19
Outline
1 Start with a simple plasma problem: 2D bump-on-tail simulation
2 Optimization by marker removal in 2D phase space
3 Generalize to 5D gyrokinetic phase space
4 Summary and possible extensions
Summary
An optimization technique by marker removal is developed for δfparticle simulations.
I The technique first constructs an importance function in theconstant-of-motion space from the linear eigenmode, then removesmarkers and scales the weights accordingly.
I The technique is very effective in simulations where δf is localizedin phase space.
I The technique can be used for a phase space of arbitrary geometryand dimension.
The technique is successfully applied to a 2D bump-on-tailsimulation and a 5D gyrokinetic TAE simulation.
I It is verified that numerical error introduced by finite number ofmarkers is mainly statistical.
I The marker removal techniqueF does not introduce apparent systematic error;F saves markers by about 75% in bump-on-tail simulation and 95% in
TAE simulation.
[Deng & Fu, Optimization by marker removal for δf particle simulations, ComputerPhysics Communications (2013)]
18/19
Possible extensionsImportance function only controls what fraction of markers is to be kept anddoes not enter simulation system directly. Choice of importance function is notunique.
Importance function can be constructed from other sources, such as analytictheories or a previous run:
Inext(zc) = maxt{Iprev(zc, t)}
Iprev(zc, t) =iprev(zc, t)
maxzc{iprev(zc, t)}
iprev(zc, t) =
∫|δf(zc,zn, t)J |dzn∫g(zc,zn, t)J dzn
≈∑j |wj(t)|S[zc,j(t)− zc]∑
j S[zc,j(t)− zc]
This is especially useful for multi-mode and nonlinear simulations.
For full-f simulations, optimal marker distribution can be obtained by directlyloading markers to be proportional to initial f , so marker removal is notneeded unless f is too complicated for direct loading.
Our marker removal optimization technique does not touch the “in-cell” part,so besides particle-in-cell, the technique can be directly applied to otherapproaches of marker particle simulations, such as particle-in-wavelet (PIW)and gridless treecode simulations.
19/19