the dd- amg solver library · the dd- amg solver library a multigrid solver library for...
TRANSCRIPT
The DD-αAMG Solver LibraryA Multigrid Solver Library for Wilson-Clover Fermions
Matthias Rottmann
contributions by: A. Frommer, K. Kahl, B. Leder, S. Krieg, A. Strebel, S. Heybrock,S. Bacchio, K. Szabo, W. Soldner and many others
Bergische Universitat Wuppertal
Lattice 2016, Southampton
Introduction Library Features Progress Report Summary & Outlook
Outline
Introduction
Library Features
Progress Report
Summary & Outlook
M. Rottmann, DD-αAMG solver library 1/14
Introduction Library Features Progress Report Summary & Outlook
Adaptive Algebraic Multigrid Approach
Two-grid error propagator for ν steps of post-smoothing
E(ν)2g = (1−MD)ν︸ ︷︷ ︸
smoother
(1− PD−1c P †D)︸ ︷︷ ︸coarse grid correction
, Dc := P †DP︸ ︷︷ ︸coarse operator
I low accuracy for D−1c and M is sufficient
I allows for introducing recursive construction for Dc
To Do: Define interpolation P and smoother M
DD-αAMG
M : Schwarz Alternating Procedure (SAP)[Hermann Schwarz 1870; Martin Luscher 2003]
P : Aggregation Based Interpolation[Brannick, Clark et al. 2010]
M. Rottmann, DD-αAMG solver library 2/14
Introduction Library Features Progress Report Summary & Outlook
The Algebraic Multigrid Principle
Smoother: 1−MD
I effective on “large” eigenvectors
I “small” eigenvectors remain
SAP
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||(I−MD)v i||/||v
i||
number i of eigenvalue λi
Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|
M. Rottmann, DD-αAMG solver library 3/14
Introduction Library Features Progress Report Summary & Outlook
Aggregation Based Interpolation
Construction
I define aggregates: domain decomposition A1, ...,As
A2
A1
A4
A3
P
P †
I calculate test vectors w1, ..., wN
I decompose test vectors over aggregates A1, ...,As
(w1, . . . , wN) = =A2
A1
As
→ P =
A1
A2
As
M. Rottmann, DD-αAMG solver library 4/14
Introduction Library Features Progress Report Summary & Outlook
The Algebraic Multigrid Principle
Coarse-grid correction: 1− PD−1c P †D
I small eigenvectors w1, . . . , wN built into interpolation P
⇒ effective on small eigenvectors
SAP
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||(I−MD)v i||/||v
i||
number i of eigenvalue λi
Aggregation + Low Modes
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||(I−PD−1
cP† D
)v i||/||v
i||
number i of eigenvalue λi
Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|
M. Rottmann, DD-αAMG solver library 5/14
Introduction Library Features Progress Report Summary & Outlook
The Algebraic Multigrid Principle
Coarse-grid correction: 1− PD−1c P †D
I small eigenvectors w1, . . . , wN built into interpolation P
⇒ effective on small eigenvectors
SAP
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||(I−MD)v i||/||v
i||
number i of eigenvalue λi
Aggregation + Low Modes
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||(I−PD−1
cP† D
)v i||/||v
i||
number i of eigenvalue λi
Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|
M. Rottmann, DD-αAMG solver library 5/14
Introduction Library Features Progress Report Summary & Outlook
The Algebraic Multigrid Principle
Two-grid method: E2g = (1−MD)(1− PD−1c P †D)
I complementarity of smoother and coarse-grid correction
I effective on all eigenvectors!
DD-αAMG
0
0.2
0.4
0.6
0.8
1
0 500 1000 1500 2000 2500 3000
||E2gv i||/||v
i||
number i of eigenvalue λi
Dvi = λivi with |λ1| ≤ . . . ≤ |λ3072|
M. Rottmann, DD-αAMG solver library 5/14
Introduction Library Features Progress Report Summary & Outlook
Adaptive Setup: How to Obtain Information about Small Eigenvectors
Inverse iteration with the method itself
start with randomtest vectors w1, . . . , wN
smooth all wia few times
build P and Dc
P good?apply method itself and
solve Dϕi = wi ∀i
set wi ← ϕi andorthonormalize w1, . . . , wN
stop
no
yes
add recurrence scheme → multilevel setup
M. Rottmann, DD-αAMG solver library 6/14
Introduction Library Features Progress Report Summary & Outlook
Performance and Setup Costs
M. Rottmann, DD-αAMG solver library 7/14
BMW-c cnfg, 3HEX-smeared 64× 643, 128 cores
30
50
200
400
100
1000
10000
−0.02−0.03−0.04−0.05mdmumcrit
mudti
me
toso
luti
on(i
nse
con
ds)
m0
mp oe BiCGStabtwo-level DD-αAMG
three-level DD-αAMGfour-level DD-αAMG
A. Frommer, K. Kahl, S. Krieg, B. Leder, R.→ arXiv:1303.1377,1307.6101
Introduction Library Features Progress Report Summary & Outlook
Performance and Setup Costs
M. Rottmann, DD-αAMG solver library 7/14
BMW-c cnfg, 3HEX-smeared 64× 643, 128 cores
30
50
200
400
100
1000
10000
−0.02−0.03−0.04−0.05mdmumcrit
mudse
tup
tim
e+
tim
eto
solu
tion
(in
seco
nd
s)
m0
mp oe BiCGStabtwo-level DD-αAMG
three-level DD-αAMGfour-level DD-αAMG
A. Frommer, K. Kahl, S. Krieg, B. Leder, R.→ arXiv:1303.1377,1307.6101
Introduction Library Features Progress Report Summary & Outlook
The DD-αAMG Solver Library
Technical Features
I C-code
I MPI + OpenMP
I SSE intrinsics
I library interface
I integrated profiling
I small parameter set forusers
I large parameter set fordevelopers
I debug information
I GNU GPL
Code Features
I Wilson+Clover
I AMG with arbitrarynumber of levels
I idling processes
I different smoothers:SAP, additive Schwarz,GMRES
I reference methods:BiCGStab, CGN,GMRES
I mixed precision,odd-even preconditioned
M. Rottmann, DD-αAMG solver library 8/14
available on github.com/DDalphaAMG
Introduction Library Features Progress Report Summary & Outlook
The DD-αAMG Library Interface
Interface Features
I conversion routines for configurations and vectors
I setup routine (initial setup)
I configuration update routine (→ HMC)
I setup update routine (→ HMC)
I mass update routines (→ Hasenbusch trick)
I solver routine (inversion for 1 RHS)
I GNU GPL → feel free to adjust it to your needs
M. Rottmann, DD-αAMG solver library 9/14
available on github.com/DDalphaAMG
Introduction Library Features Progress Report Summary & Outlook
Progress Report: Existing Branches
DD-αAMG for KNC/KNL
I SAP smoother implemented from scratch
I coarse grid correction from DD-αAMG
I DD-αAMG inherited SIMD vectorization from KNC-code
S. Heybrock, D. Richtmann, P. Georg, T. Wettig, R.→ arXiv:1512.04506,1601.03184
DD-αAMG for twisted mass fermionsI the interpolation P in DD-αAMG respects γ5P = Pγc5=⇒ D + iµγ5 Dc + iµγc5
S. Bacchio, J. Finkenrath, C. Alexandrou, A. Frommer, K. Kahl, R.→ Talk by S. Bacchio
M. Rottmann, DD-αAMG solver library 10/14
Introduction Library Features Progress Report Summary & Outlook
Progress Report: Future Branch – Eigensolver for γ5D
48× 243 CLS cnfg, mπ ≈ 290MeV
I Jacobi-Davidson with amultigrid solver
I outperforms PARPACK for400 eigenpairs
I improved scaling with latticevolume
100
1000
10000
50 100 150 200 250 300 350 400
solv
eti
me
(in
core
-h)
number of eigenpairs (λ, vλ)
PARPACKJacobi-Davidson
A. Strebel, J. Simeth, G. Bali, S. Collins, A. Frommer, K. Kahl, I. Kanamori,B. Muller, R.→ arXiv:1509.06865
M. Rottmann, DD-αAMG solver library 11/14
Introduction Library Features Progress Report Summary & Outlook
Outlook
I eigensolver: improve algorithm(clever restart strategies, etc.)=⇒ publication
100
1000
10000
50 100 150 200 250 300 350 400
solv
eti
me
(in
core
-h)
number of eigenpairs (λ, vλ)
PARPACKJacobi-Davidson
I overlap solver: prepare code for publicationJ. Brannick, A. Frommer, K. Kahl, B. Leder, A. Strebel, R.→ arXiv:1410.7170
−2
−1
0
1
2
−1 0 1 2 3 4 5 6 7
imag
inar
yax
is
real axis
spec(DW )spec(DN )
spec(DND−1W )
I vectorization: generalization, integrate AVX
Instruction Unit
ALU ALU . . . ALU
Mem Mem . . . Mem
M. Rottmann, DD-αAMG solver library 12/14
Introduction Library Features Progress Report Summary & Outlook
Summary & Outlook
M. Rottmann, DD-αAMG solver library 13/14
DD-αAMGSFB/TR55
Wilson Solver
OverlapSolver
Eigensolverfor γ5D
DynQCDBMW-c
KNC-codefor QPACE2/3
OpenQCDLuscher/CERN
Twisted MassHPC-LEAP
github.com/DDalphaAMG