DOE HEP Physics Program ReviewJune 14-16, 2005 @SLAC

Advanced Computations Department

Kwok Ko

* Work supported by U.S. DOE ASCR & HEP Divisions under contract DE-AC02-76SF00515

ACD Mission

Develop new simulation capability to support accelerator R&D at SLAC & accelerator facilities across SC,

Advance computational science to enable ultra-scale computing on SC’s flagship computers (NERSC, ORNL)

Share resources with community and educate/train future computational scientists.

Support: Base program, SciDAC, Accelerator projects, SBIR + others

Personnel:15 people/13 FTE (5 computational physicists, 7 computer scientists, 2 graduate students, 1 admin/technical assistant)

Output: 3 PhD thesis, 5 papers, 3 reports, 30 talks/posters (2003-05)

Formed in 2000 to focus on high performance computing with the mission to:

ACD R&D Overview & SciDAC

Simulation and Modeling

ComputationalScience

Parallel Code Development

HHigh Performance Computing (NERSC, ORNL)

Accelerators

SLACFNALANLJlabMITDESYKEKPSI

ACD

Accelerator Modeling Computational Mathematics

Computing Technologies

SciDAC

LBNLLLNLSNLStanford UCDRPI, CMUColumbiaUWisconsin

DSBIR - STAR Inc

Modeling and Simulation

Electromagnetic Modeling

NLC Cell Design

Refinement Performance Optimization Visualization

CAD/Meshing Partitioning Solvers Analysis

Elements of Computational Science

Large-scale electromagnetic modeling is enabled by advancing all elements through SciDAC collaborations

SciDAC ESS Team

ISICs (TSTT, TOPS, PERC) and SAPP

UCD

K. Ma, H. Yu Z. Bai

RPI

M. Shephard, A. Brewer, E. Seol

SNL

P. Knupp, K. Devine.L. Fisk, J. Kraftcheck

LBNL

E. Ng, W. Gao, X. Li, C, YangP. Husbands, A. Pinar,D. Bailey, D. Gunter

LLNL

L. Diachin, D. Brown, D. Quinlan, R. Vuduc

Stanford

G. Golub

Columbia

D. Keyes

CMU

O. Ghattas V. Akcelik

UWisconsin

T. Tautges, H. Kim,

Computational Mathematics

L. Lee, L. Ge, E. Prudencio, S. Chen (Stanford),

Accelerator Modeling

K. Ko, V. Ivanov, A. Kabel, Z. Li, C. Ng,, L. Xiao, A. Candel (PSI)

Computing Technologies

N. Folwell, G. Schussman, R. Uplenchwar, A. Guetz (Stanford)

SLAC/ACD

“Electromagnetic Systems Simulation”

Parallel Code Development

Electromagnetics(SciDAC funded)

Beam Dynamics(SLAC supported)

“Unstructured Grid and Parallel Computing”

Omega3PTau3P/T3P S3P

Time Domain Simulation

With Excitations

Frequency DomainMode Calculation

Scattering Matrix Evaluation

Finite-Element Discretization

Track3P – Particle Tracking with Surface Physics

GeneralizedYee Grid

V3D – Visualization/Animation of Meshes, Particles & Fields

Weak-strongBeam-beam

Strong-strong Beam-beam

TrafiC4 - CSR

Achievements in Accelerator Science(Electromagnetics & Beam Dynamics)

Omega3P: Sum over eigenmodes

NLC DDS Wakefields

NLC 55-cell DDS

Omega3P/Tau3P computed the long-range wakefields in the 55-cell Damped Detuned Structure to verify the NLC design in wakefield suppression by damping and detuning.

Tau3P: Direct beam excitation

Tau3P: Direct beam excitation

Omega3P Wakefields Tau3P Wakefields

Tau3P: Direct beam excitation

NLC Dark Current

Dark current @ 3 pulse risetimes

Track3P Data

-- 10 nsec-- 15 nsec-- 20 nsec

Dark current pulses were simulated for the 1st time in a 30-cell X-band structure with Track3P and compared with data. Simulation shows increase in dark current during pulse risetime due to field enhancement from dispersive effects.

Track3P: Dark current simulation

Red – Primary particles, Green – Secondary particles Red – Primary particles, Green – Secondary particles

Track3P: Dark current simulation

ILC Cavity Design

An international collaboration (DESY, KEK, SLAC, FNAL, Jlab) is working on a Low-Loss cavity (23% lower cryogenic loss) as a viable option for the ILC linac. SLAC is calculating the HOM damping & multipacting for the DESY and KEK designs.

ILC LL 9-cell Cavity Design

Qext in ICHIRO-2 cavity

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1.45E+09 1.55E+09 1.65E+09 1.75E+09 1.85E+09 1.95E+09

F (GHz)

Qext

ILC Cavity HOM Damping Partitioned Mesh of LL Cavity

Complex Omega3P is being used to calculate the Qext of dipole modes in the DESY and KEK LL cavity designs.

DESY KEK

PEP-II Vertex Bellows Damping

Ceramic tile absorber Bellows mode Dielectric loss

Omega3P was used to study the effectiveness of ceramic tiles mounted on the bellows convolution to damp localized modes that contribute to HOM heating of the bellows. Bellows modes can be damped to very low Qs (~20-50)

Bellows Modes

PEP-II Vertex Bellows

LCLS RF Gun Cavity Design

ACD provided the dimensions for the LCLS RF Gun cavity that meet two important requirements:

minimized dipole and quadrupole fields via a racetrack dual-feed coupler design,

reduced pulse heating by rounding of the z coupling iris.

-0.003

-0.002

-0.001

0.000

0.001

0.002

0.003

-200 -100 0 100 200

rf phase (degree)

cylindrical cavity

racetrack with offset=0.05 "

Qu

ad

(

βr)

/mm

A new parallel Particle-In-Cell (PIC) capability is being developed in T3P for self-consistent modeling of RF guns needed for the LCLS upgrade, future light sources and FELs.

Quad

LCLS CSR Effects

LCLS Bunch Compressor (with P. Emma): Predict FEL performance in the self-consistent Coherent Synchrotron Radiation (CSR) regime for different compressor settings

Coherent Edge Radiation: Field viewer module for TraFiC4 allows study of the spatial & temporal behaviour of the detector signal

Slice Saturation Power Slice Gain Length

Tevatron Beam-Beam Simulation

Tevatron (with Y. Cai and T. Sen): Calculate actual lifetimes and lifetime signatures for the machine at injection and collision for different machine parameters

New version of parallel beam-beam framework PLIBB:• Allows billions of particle-turns• Resolves ~100h lifetime (collision case!)• Handles chromaticity exactly • Strong-strong being integrated

Lifetime enhancement with lowered chromaticity

Example result - Low particle loss rates at collision

PLIBB Results

PSI Cyclotron HOM Analysis

1st ever eigenmode analysis of an entire ring cyclotron as part of a PhD research (L. Stingelin) to investigate the beam-cavity interactions in the existing machine and future upgrade.

CAVITY VACUUM CHAMBER MIXED MODES (NEW)

Advances in Computational Science (SciDAC)

Parallel Meshing (SNL, UWisconsin)

Processor: 1 2 3 4

To be able to model multiple ILC cavities a parallel meshing capability has been developed in collaboration with SNL and UWisconsin (PhD thesis) to facilitate the generation of VERY LARGE meshes on the supercomputer directly to overcome the memory limitation of desktops.

Omega3P

Lossless Lossy Material

PeriodicStructure

ExternalCoupling

ESILISIL w/ refinement

Implicit RestartedArnoldi SOAR Self-Consistent

Loop

WSMP MUMPS SuperLU Kryov Subspace Methods

Domain-specific preconditioners

Eigensolvers (LBL, UCDavis, Stanford) With LBL, UCD and Stanford, a comprehensive capability has been under development for solving large, complex RF cavities to accuracies previously not possible. The parallel eigensolver Omega3P has been successfully applied to numerous accelerator cavities and beamline components.

Mesh Refinement (RPI)

In modeling RIA’s RFQs, Adaptive Mesh Refinement (AMR) provided accuracy gain of 10 and 2 in frequency and wall loss calculations with Omeg3P over standard codes, while using a fraction of CPU time compared to the case without AMR. Wall Loss on AMR Mesh

More accurate f and Q predictions reduce the number of tuners and tuning range, and allow for better cooling design

RFQ - Frequency Convergence

54.354.454.554.654.754.854.9

5555.155.2

0 1000000 2000000 3000000 4000000

Number of Unknowns

Fre

qu

ency

in

MH

z

RFQ - Q Convergence

5750

5800

5850

5900

5950

6000

6050

6100

0 1000000 2000000 3000000 4000000

Number of Unknowns

Q

AMR speeds up convergence thereby minimizing computing resources

FrequencyConvergence

Qo Convergence

Omega3PSensitivity

meshingsensitivity

optimizationgeometrigeometri

ccmodelmodel

Omega3P meshingmeshing

(only for discrete sensitivity)

Shape Optimization (CMU, SNL, LBNL)

An ongoing SciDAC project is to develop a parallel shape optimization tool to replace the existing manual process of optimizing a cavity design with direct computation. The capability requires the expertise from SciDAC’s ISICs.

Visualization (UCDavis)

New graphics tools for rendering LARGE, multi-stream, 3D unstructured data have been developed, to be supported by a dedicated visualization cluster to help in analyzing cavity design, such as mode rotation in the ILC cavity.

Graphics tools for rendering LARGE, 3D multi-stream, unstructured data have been developed and a visualization cluster soon be installed, both to support accelerator analysis

Mode rotation (in space and time) exhibited by the two polarizations of a damped dipole mode in ILC cavity

Dissemination HEP/SBIR: STAR Inc and ACD are developing the GUIs to interface SLAC’s parallel codes which are in use at e.g. FNAL and KEK. These codes potentially can replace use of commercial software (MAFIA, HFSS) at DOE sites to save costs ~million+ $ per year in leases.

USPAS: SciDAC codes and capabilities are shared regularly with the community via the course “Computational Methods in Electromagnetism”

USPAS sponsored by theCornell University held in Ithaca, NY

June 20 - July 1, 2005http://uspas.fnal.gov/

Education/Training

PhDs completed in ACD;

Yong Sun, SCCM, Stanford University, March 2003“The Filter Algorithm for Solving Large-Scale Eigenvalue Problems from Accelerator Simulations”

Greg Schussman, Computer Science, UCDavis, December 2003“Interactive and Perceptively Enhanced Visualization of Large, Complex Line-based Datasets”

Lukas Stingelin, Physics, Ecole Polytechnique Lausanne, December 2004“Beam-cavity Interactions in High Power Cyclotrons”

PhDs in progress;

Adam Guetz, ICME, Stanford University

Sheng Chen, ICME, Stanford University

Summer interns – Grad/Undergrad

ACD Goals Continue to support Accelerator Science across SC Continue SciDAC collaborations in Computational Science Involve in Astroparticle Physics & Photon Science

ILC

BPM & Wakefields in LCLS Undulator

XFEL SC RF Gun MIT PBG

Cavity for Jlab 12 GeV Upgrade

ILC LL Cavity & Cryomodule