A Science Gateway for Atomic and Molecular Physics

Barry I Schneiderlowast

National Institute of Standards and Technology Gaithersburg Maryland 20899 USA

Klaus Bartschatdagger and Oleg ZatsarinnyDagger

Drake University Des Moines IA 50311 USA

Igor Braysect

Curtin University Perth GPO Box U1987 Western Australia

Armin Scrinzipara

Ludwig-Maximilians Universitat Munchen Germany

Fernando Martınlowastlowast and Markus Klinkerdaggerdagger

Universidad Autonoma de Madrid Catoblanco Madrid 28049 Spain

Jonathan TennysonDaggerDagger

Department of Physics and Astronomy University College LondonGower Street London WC1E 6BT United Kingdom

Jimena D Gorfinkielsectsect

School of Physical Sciences The Open University Milton Keynes MK7 6AA United Kingdom

Sudhakar Pamidighantamparapara

Indiana University CIB 2709 E 10th Street BloomingtonIn 47408 and the eXtreme Science and Engineering Discovery Environment(XSEDE)

(Dated January 9 2020)

We describe the creation of a new Atomic and Molecular Physics science gateway (AMPGateway)

NoteThis paper was supposed to appear in an ACM transactions of PEARC19 It wasaccidentally omitted

The gateway is designed to bring together a subset of the AMP community to work collectivelyto make their codes available and easier to use by the partners as well as others By necessity aproject such as this requires the developers to work on issues of portability documentation ease ofinput as well as making sure the codes can run on a variety of architectures Here we outline ourefforts to build this AMP gateway and future directions

PACS numbers

I INTRODUCTION

On May 14-16 2018 an NSF supported workshopentitled ldquoDeveloping Flexible and Robust Software inComputational Atomic and Molecular (AampM) Physicsrdquowas organized by Barry Schneider (chair) Robert Forrey

lowastElectronic address barryschneidernistgovdaggerElectronic address klausbartschatdrakeeduDaggerElectronic address oleg˙zoiyahoocomsectElectronic address igorbraycurtineduauparaElectronic address arminscrinzilmudelowastlowastElectronic address fernandomartinuamesdaggerdaggerElectronic address markusklinkergmailcomDaggerDaggerElectronic address jtennysonuclacuksectsectElectronic address jimenagorfinkielopenacukparaparaElectronic address pamidigsiuedu

(Penn State) and Naduvalath Balakrishnan (UNLV)at the Institute for Theoretical Atomic and MolecularPhysics Harvard-Smithsonian ITAMP [5] The purposeof the workshop was to bring together a group of interna-tionally known researchers in computational atomic andmolecular physics to

bull Identify and prioritize outstanding problems inAampM science which would benefit from a concertedcommunity effort in developing new software toolsand algorithms that would lead to more rapid andproductive scientific progress for the entire commu-nity

bull Discuss approaches to optimize achieving that goal

bull Produce and disseminate a report of the workshopto the community

arX

iv2

001

0228

6v1

[ph

ysic

sco

mp-

ph]

7 J

an 2

020

2

A concerted community effort is underway to developand maintain these tools in order to ensure continuedscientific progress The group acknowledged that incontrast to some other communities AampM physics haslagged behind in developing community software that isrobust and can be used in a relatively straightforwardway by other than the group who developed that soft-ware While there are exceptions many software pack-ages are poorly documented poorly written and onlyusable by a set of local ldquoexpertsrdquo The tools themselvesare capable of treating scientific and technologically in-teresting problems but they are only accessible to a smallgroup of people The codes are not always maintainedand the lack of coordination among the developers hasled to a lot of ldquoreinventing the wheelrdquo The group feltstrongly that the efforts being expended in developingthese computational tools should be available and usableby future generations of AampM scientists

The success of the workshop led six of the groups towork together and develop an XSEDE proposal to buildand maintain a Science Gateway devoted to the codesdeveloped in these groups That proposal was supportedand since May of 2018 there has been decent progress Anumber of the codes are already ported and running onvarious XSEDE platforms Some progress has been madein making them usable by others within the group butnot yet the outside world We are now taking steps toachieve this last goal

The AMPGateway uses the multi-tenanted ApacheAiravata middleware framework [2ndash4] served by the Sci-GaP hosting services for sustained operation In thefirst stage of our efforts the software suites have beendeployed as independent applications with specific inputinterfaces Community building has already started anda few additional software suites have been identified forinclusion in phase two The interoperability of the soft-ware suites is very important and will be addressed as afollow-on

The present manuscript is divided into four major sec-tions In the Introduction we provide a history of howand why the project got started and our decision to goto XSEDE [1] for support for the gateway In Section IIwe present some information on the AMP codes that arealready available on the gateway Section III is devotedto the details of the construction and deployment of thegateway In Section IV we discuss issues of broadeningusage of the gateway and questions of community build-ing

II CURRENT CODE STATUS

At present we have concentrated our major effort onfive codes A brief description of these packages is givenbelow

A BSR

The B-spline R-matrix (BSR) method and the ac-companying computer code [6] were developed by OlegZatsarinny in the group of Klaus Bartschat at DrakeUniversity The program computes transition-matrix el-ements for electron collisions with atoms and ions as wellas photoionization processes From these cross sectionsand other experimentally observable parameters can beobtained The code can also be run in a mode that pro-vides atomic structure information through energy levelsand oscillator strengths

The BSR approach is a particular variant of the R-matrix method to solve the close-coupling equations incoordinate space In this respect it is complementary tothe convergent close-coupling (CCC) approach describedbelow BSR is an alternative formulation of the well-known R-matrix code developed in Belfast under thelong-term leadership of Philip Burke The Belfast codeis somewhat singular in that it is readily available andused by a small group of users While the last generalwrite-up appeared in 1995 [7] updated versions are avail-able [8] A comprehensive introduction to R-matrix the-ory for atomic and molecular collisions processes as wellan overview of many applications can be found in thebook by Burke [9]

The published BSR code [6] is a serial version whichwas written in the non-relativistic and semi-relativistic(Breit-Pauli) frameworks Relativistic (DBSR) and MPI-parallelized versions as well as extensions to treat ioniza-tion processes (similar to the CCC method described be-low) exist and are being used by the developer and a smallgroup of collaborators Executables of the parallelizedcodes (currently running on Stampede2) will be uploadedto the Gateway in the near future The BSR and DBSRpackages are a prime example where updated documen-tation and a wide distribution are urgently needed be-fore critical expertise is lost Fortunately the urgencywas recently recognized by the NSF and resulted in thefunding of a three-year proposal to achieve exactly thesegoals We expect the gateway described in the presentpaper to be one of the vehicles to ensure significant futureprogress

A comprehensive overview of the BSR method and itsapplications at the time was published by Zatsarinny andBartschat [10] The most noteworthy features of the codeare

bull Use a finite-element (B-spline) rather than a finite-difference approach in the calculation of the matrixelements needed to set up the hamiltonian in theinner region

bull Employ non-orthogonal sets of one-electron orbitalsto account for the term-dependence of the valenceorbitals in particular for complex open-shell tar-gets thereby providing an economical and accuratedescription of the target states and much flexibilityin building the scattering wavefunction as well as

3

pseudostates to further improve the target descrip-tion and enable the treatment of electron-impactsingle-ionization as well as photon-driven doubleionization processes

The BSR code has the following major parts

bull Build the Nminus and (N+1)minuselectron configurations

bull Generate all necessary one-electron and two-electron matrix elements to set up the target andscattering Hamiltonians in the internal region

bull Diagonalize these Hamiltonians

bull Propagate the wavefunction from the R-matrixboundary r = a to ldquoasymptotiardquo (rb) where it canbe matched to known analytic forms The propaga-tion requires the solution of a set of coupled differ-ential equations using known long-range potentialsand needs to be repeated for each scattering en-ergy If angle-differential ionization processes withtwo free electrons in the final state are to be treatedas well the inner region may need to be increasedbeyond the original criterion

Even though there is no general way to predict wheremost of the computational effort is needed in most casesthe generalized eigenvalue problem (diagonalization withall eigenvalues needed) of the (N + 1)minuselectron Hamilto-nian is a very time-consuming step For complex targetssetting up this Hamiltonian can be expensive as wellFor ionic targets the wealth of resonances may requiremany thousands of collision energies to be treated whichcan result in significant time going into the asymptoticregion

To summarize The BSR method is closely relatedto both the Belfast R-matrix approach and the CCCmethod described below The two R-matrix codes weredesigned to handle complex targets and many energieswhile the CCC code can handle more processes but isessentially limited to quasi-one and quasi-two electrontargets Some benchmark comparisons for problems thatall three methods should be able to handle have been per-formed As expected the results are numerically equiva-lent but one or the other method may be more efficientThe details strongly depend on the complexity of the tar-get and the energies for which results are required

B CCC

The original implementation of the Convergent Close-Coupling method was designed to produce accurate crosssections for scattering of light projectiles from quasi one-and two-electron targets [14] It began with electron-hydrogen scattering [15] and was further extended toquasi-one electron targets well-modeled by one valenceelectron above a frozen Hartree-Fock core [16] It wasthen extended to the helium target [17] and quasi two-electron targets such as Be [18] The main features are

bull Expansion of the target state in a sufficiently com-plete L 2 basis size N to treat cases where bothexcitation and ionization of the target are possiblewith convergence tested by systematically increas-ing N

bull Expansion of the scattering wavefunction in themomentum based Lippmann-Schwinger equation

bull Introduction of numerical quadrature to reduce theproblem to a very large set of linear algebraic equa-tions

CCC has been extended to positron scattering wherethe positron introduces a second center capable of form-ing the electron positron bound state known as positro-nium (Ps) [19] This is an example of a rearrangementcollision and as such introduces even more complexityinto the computational procedure Such calculations canbe ldquotime-reversedrdquo to be considered as Ps scattering on(anti)protons to form (anti)hydrogen [20] A review ofthe CCC method for positron scattering has been givenby Kadyrov and Bray [21]

On the computational side the CCC codes have beenparallelized to use OpenMP on a node and MPI betweennodes and have been deployed successfully on Comet andStampede2 A GPU implementation currently under-way shows immense promise with up to two orders ofmagnitude speedup

C UKRMol(+)

The UK Molecular R-matrix codes were designed totreat low-energy elastic and inelastic electron-moleculecollisions using the R-matrix method they have evolvedto also study photoionization and positron-molecule col-lisions as well as to produce the input required for time-dependent molecular R-matrix with time dependence(RMT) calculations [22] Similar to BSR they are basedon the R-matrix method and the general theory can befound in the book by Burke [9]

The now frozen release version of the (mainly serial)UKRMol suite uses Gaussian Type Orbitals (GTOs) torepresent both the target and continuum orbitals A pub-lication presenting this code by Carr et al [23] followed aproject which substantially updated (to Fortran95) andstandardized the programming used particularly in theinner region A review article by Tennyson [25] from thesame period gives a comprehensive overview of theoryused and the functionality of the code

The use of GTOs to represent the continuum leads toconstraints on both the size of the inner region that canbe used and the free electron energy range Recently anew suite known as UKRMol+ has been developed ledby Zdenek Masın and Jimena Gorfinkiel [23] The codeuses the new GBTOlib integral library to determine allthe one- and two-electron integrals needed in the mixedbasis of GTOs and B-splines it offers the choice of using

4

GTOs B-splines or hybrid GTOs ndash B-splines to representthe continuum the bound orbitals are always describedby GTOs The library which uses object oriented fea-tures from the Fortran2003 standard involves distributedand shared-memory parallelization UKRMol+ incorpo-rates a number of algorithmic improvements including afaster configuration state function (CSF) generation andparallization of the construction and diagonalization ofthe N and (N + 1) Hamiltonians [29] Further paral-lelization to avoid IO to disk during the evaulation oftransition moments for photoionization and RMT inputis currently being tested

Both suites contain a rich array of outer region func-tionality including automated resonance detection andfitting bound state detection computation of multichan-nel quantum defects rotational excitation and evaluationof photoionization cross sections So far applications ofthe UKRMol+ suite are limited [26ndash28] but a full releaseand associated article will be available shortly [29]

The codes have been available as freeware for morethan a decade and are widely used the software canbe downloaded as a tarball and installed (in the case ofUKRmol+) using cmake provided the necessary librariesare available in the system Neither suite is straightfor-ward to use without training Quantemol-N [30] is a com-mercial front end which has led to a significant increase inthe user base of the code A set of perl scripts developedby Karel Houfek that simplify writing the input is alsonow available both for electron scattering and photoion-ization calculations Further details can be found on thewebsite of the UK Atomic Molecular and Optical physicsR-matrix consortium (httpswwwukamorcom)

D tRecX

The tRecX code package [31 32] is a general frameworkfor solving initial value problems of the form

part

parttΨ = D[Ψ t] + Φ(t) (1)

for an arbitrary number of spatial dimensions and a vari-ety of coordinate systems The main design is for linearD but non-linear operators can also be used

1 Applications

The code has been primarily used for solving the time-dependent Schrodinger equation of atomic and molecularsystems in ultra-short pulses and in strong near-IR fieldsThe most significant results are fully differential spectrafor single- and double photo-emission from the He atomat near infrared wave-length [33] strong field ionizationrates of noble gases [34] and differential spectra for smalldi- and tri-atomic molecules [35 36] cf Fig 1

FIG 1 Left Helium in full dimensions double emissioncross-section σ(pz1 pz2) for a 2-cycle pulse at wavelength 400nm and 5times1014 Wcm2 intensity Anti-correlated emission isfavored Right haCC calculation for CO2 photo-emission byan 800 nm laser pulse at intensity 1014 Wcm2 up to energies25 au in the xz-plane at 45 alignment of the molecular axisto polarization direction (from [36])

2 Methods

The code uses three newly developed key techniques

1 irECS mdash ldquoInfinite range exterior complex scal-ingrdquo [37] as absorbing boundary conditions and forthe computation of life-times irECS is a variantof exterior complex scaling with an infinitely wideboundary for absorption at all wave lengths

2 tSurff mdash the time-dependent surface fluxmethod [38] for photo-emission spectra BytSurff the actual numerical solution remainscontained in a small reactive region of typically 20to 100 atomic units

3 haCC mdash the ldquohybrid anti-symmetrized CoupledChannelsrdquo method [39] combines Gaussian-basedneutral and ionic CI states with a numerical single-electron basis The numerical basis extends overthe whole system thus ensuring the proper descrip-tion of energetic electron-molecule collisions

Two of these techniques are reflected in the codersquos nametRecX = tSurff + irECS

3 Structure and inputs

An effort was made to keep the code flexible with-out sacrificing efficiency Systems with dimensions fromone (popular models) to six (He in elliptically polarizedfields) as well as multi-channel models (photo-electronspectra for molecules) are treated within the same frame-work the degrees of freedom map into a tree hierarchyinducing a tree-structure for vectors and operators andproducing transparent and efficient code by recursion

Basis functions are arranged in a tensor-tree with avariety of basis sets finite-elements FE-DVR and gridsthat can be combined on any number of coordinate axeswith multiple basis sets on the same axis Discretizationand operators can all be input-controlled For example

5

define BOX 20Axis namenCoefflowerupperfuncsorderPhi5expImEta3-11 assocLegendrePhiRn60 0BOXpolynomial15Rn20 BOXInftypolExp[05]

with Eta for cos θ defines polar coordinates The bases

exp(imφ) and P|m|l (cos θ) combine to the spherical har-

monics up to l = 2 and the r-coordinate uses 6015=4FE-DVR elements on [0 20] with 15 polynomials on eachand 20 exponentially damped polynomials at the endAn example for operator specification is

Operator hamiltonian05(ltd_1_dgtlt1gt+lt1gtltd_1_dgt+ltQQgtlt1gt+lt1gtltQQgt)

for (larrminuspart xminusrarrpart x +

larrminuspart yminusrarrpart y + x2 + y2)2 Many standard oper-

ators are pre-defined for various coordinate systems

4 Software

The code is open source hosted on a Gitlab reposi-tory [32] It is written mostly in C++ and linked withsome Fortran-based libraries Optionally functionalitycan be extended by linking FFTW and GSL Standardcompilers are gcc and Intel ports to Windows and MacrsquosClang have been successful but are not actively sup-ported Compilation is through CMake and Doxygendocumentation is available Tutorials and further mate-rials are available in the git and from the tRecX web-site [31]

E XChem

XChem [40 41] is a solution for an all-electron ab initiocalculation of the electronic continuum of molecular sys-tems XChem combines the tools of quantum chemistry(as implemented eg in MOLCAS [44]) and scatteringtheory to accurately account for electron correlation inthe single-ionization continuum of atoms [41 43] smalland medium-size molecules [42] At its core lies a close-coupling expansion combined with the use of a hybridGaussian and B-Spline basis set [40]

This approach yields the scattering states of the molec-ular system via the eigenstates of the close-coupling ma-trix (CCM) While useful in their own right the full po-tential lies in using the close-coupling matrix as a startingpoint for time-dependent calculations In doing so onemay explicitly model the interaction of molecules withultrashort (attosecond) pulses The large band widthsof such pulses lead to the coherent excitation of multi-ple ionization channels whose coupling (accurately de-scribed in XChem) gives rise to complex phenomena

An attractive feature of XChem is that the architec-ture of the basis functions (Fig 2) and the use of MOL-CAS allow one to describe the electronic continuum of

FIG 2 Ilustration of the XChem basis architecture inbenzene Cyan B-splines mangenta Gaussians at the cen-ter of mass of the molecule blue and black Gaussians at theatomic sites not overlapping with B-splines

medium-size molecules at the same level of theory asmulti-reference CI methods do for the ground and thelowest excited states of such molecules At present thelargest systems treated have of the order of ten atoms

1 What can XChem do

XChem can compute

bull The CCM for a user-defined set of ionization chan-nels (each defined as an ionized molecular state cou-pled to electrons of given angular momenta) and in-cluding short range states relevant to the problemat hand

bull Scattering states and scattering phases by asymp-totic fitting to the analytical solution

bull Total and partial photoionization cross sectionswithin perturbation theory

bull Lifetime and character of resonances embedded inthe molecular continuum either via analysis of thecross section or via inclusion of a complex absorbingpotential in the CCM yielding complex eigenener-gies

bull The electron dynamics during and after ionizationcaused by and probed with ultrashort laser pulsesby solving the time-dependent Schrodinger equa-tion using the CCM

bull The angular distribution of photo electrons (inprogress)

2 What can XChem be used for

XChem is a valuable tool for

bull The theoretical study of ultrafast processes in inAttosecond pump-probe experiments AttosecondTransient Absorption Spectroscopy (ATAS) and

6

Reconstruction of Attosecond Beatings by Interfer-ence of Two-Photon Transitions (RABBIT)

bull The investigation of photoionization of complexmolecules close to threshold where electron corre-lation effects are crucial to describe the photoion-ization cross sections

bull The study of ionization processes intrinsically de-pendent on electron correlation like autoionizationand Auger decay

bull The computation of potential energy surfaces forthe investigation of molecular dynamics during andafter fast photoionization events

3 Who is using XChem

bull Researchers in (computational) quantum chemistryor molecular physics interested in studying electrondynamics in the ionization continuum of molecules(eg photoionization charge migration etc)

bull Laboratories investigating ultrafast phenomenain many-electron atoms small and medium-sizemolecular systems

F Other Interesting Software

We are also standing up the rather old Many Elec-tron Structure Applications (MESA) code that was de-veloped at Los Alamos in the 1980rsquos and modified tocompute electron scattering and photoionization crosssections from polyatomic molecules using the ComplexKohn Method [45] While this code is old and in need ofsignificant modernization it was built to compute elec-tron scattering and photoionization transition matrix el-ements for general polyatomic molecules a capability notpresent in other codes and of interest to many users

There is also an effort underway to incorporateMOLSCAT [46] a heavy particle collision code forvibrational-rotational scattering in molecules Table II Fsummarizes some of the applications deployed

III AMP SCIENCE GATEWAY DEPLOYMENTAND APPLICATION INTEGRATION

The AMP Gateway is deployed using the Apache Aira-vata Framework [4] It relies on the Science GatewayPlatform as a Service (SciGaP) (httpsscigaporg)hosting services [3] at Indiana University The Sci-GaP platform provides gateway services using the multi-tenanted Apache Airavata middleware The Airavatacore enables features such as managing user identityaccounts authorization provisioning and the ability to

access XSEDE and other high performance computa-tional resources such as XSEDErsquos Stampede2 Comet andBridges These resources are transparently integratedinto the AMP gateway and the batch queues are usedfor scheduling the execution of models using applicationsand user defined parameters

A User Accounts Authentication andAuthorization

The AMP gateway user accounts can be created byusers by providing a userid and setting a password alongwith providing email for verification process In addi-tion to this an automated process using the user insti-tutional login via CI-Logon [47] is also provided whichavoids the email verification step The gateway adminis-trator controls the access to the resources and needs toapprove the user for gateway resource access The usersget a ldquogateway pending rolerdquo when they register Thegateway middleware provides authentication and autho-rization services through the Keycloak [2] identity man-agement system supported by SciGaP Once the gatewayadministrator provides approval the role of the new userwill be changed to ldquogateway-userrdquo which enables accessto gateway resources and applications The gateway ad-ditionally provides ldquoadmin-read-onlyrdquo and ldquoadmin rolesrdquowith their associated permissions for a group to sharethe admin responsibilities and reporting purposes Aldquogateway-userrdquo then can use gateway services such ascreating monitoring sharing and cloning experiments(computational simulations) The users can also addtheir own compute resource allocations using the func-tions available under ldquoUser Settingsrdquo The ldquoadminrdquo usershave the authorization to control metadata for accessingXSEDE through the gateway ldquocommunity loginrdquo regis-ter and deploy applications and their (user) interfacesmanage users and monitor and access all user experi-ments These privileges enable the admin to efficientlytroubleshoot any issues relating to the user services Theldquoadmin-read-onlyrdquo users can view all ldquoadminrdquo related in-formation but not modify any of the settings

B AMP Application Deployment and UserInterface Creation

The AMP gateway started with four specific applica-tions described above BSR XChem CCC and tRecXsuites These applications were compiled and tested on anumber of the XSEDE resources Each of them requiresa different set of libraries and in the case of XChem inte-gration with other Open source software such as Open-Molcas [12] They were independently tested by the col-laborating XSEDE ECSS consultant when deployed bythe developers or deployed by the consultant to estab-lish the required environments and tweak makecmakedata The Application deployment in the AMP Gate-

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

3

pseudostates to further improve the target descrip-tion and enable the treatment of electron-impactsingle-ionization as well as photon-driven doubleionization processes

The BSR code has the following major parts

bull Build the Nminus and (N+1)minuselectron configurations

bull Generate all necessary one-electron and two-electron matrix elements to set up the target andscattering Hamiltonians in the internal region

bull Diagonalize these Hamiltonians

bull Propagate the wavefunction from the R-matrixboundary r = a to ldquoasymptotiardquo (rb) where it canbe matched to known analytic forms The propaga-tion requires the solution of a set of coupled differ-ential equations using known long-range potentialsand needs to be repeated for each scattering en-ergy If angle-differential ionization processes withtwo free electrons in the final state are to be treatedas well the inner region may need to be increasedbeyond the original criterion

Even though there is no general way to predict wheremost of the computational effort is needed in most casesthe generalized eigenvalue problem (diagonalization withall eigenvalues needed) of the (N + 1)minuselectron Hamilto-nian is a very time-consuming step For complex targetssetting up this Hamiltonian can be expensive as wellFor ionic targets the wealth of resonances may requiremany thousands of collision energies to be treated whichcan result in significant time going into the asymptoticregion

To summarize The BSR method is closely relatedto both the Belfast R-matrix approach and the CCCmethod described below The two R-matrix codes weredesigned to handle complex targets and many energieswhile the CCC code can handle more processes but isessentially limited to quasi-one and quasi-two electrontargets Some benchmark comparisons for problems thatall three methods should be able to handle have been per-formed As expected the results are numerically equiva-lent but one or the other method may be more efficientThe details strongly depend on the complexity of the tar-get and the energies for which results are required

B CCC

The original implementation of the Convergent Close-Coupling method was designed to produce accurate crosssections for scattering of light projectiles from quasi one-and two-electron targets [14] It began with electron-hydrogen scattering [15] and was further extended toquasi-one electron targets well-modeled by one valenceelectron above a frozen Hartree-Fock core [16] It wasthen extended to the helium target [17] and quasi two-electron targets such as Be [18] The main features are

bull Expansion of the target state in a sufficiently com-plete L 2 basis size N to treat cases where bothexcitation and ionization of the target are possiblewith convergence tested by systematically increas-ing N

bull Expansion of the scattering wavefunction in themomentum based Lippmann-Schwinger equation

bull Introduction of numerical quadrature to reduce theproblem to a very large set of linear algebraic equa-tions

CCC has been extended to positron scattering wherethe positron introduces a second center capable of form-ing the electron positron bound state known as positro-nium (Ps) [19] This is an example of a rearrangementcollision and as such introduces even more complexityinto the computational procedure Such calculations canbe ldquotime-reversedrdquo to be considered as Ps scattering on(anti)protons to form (anti)hydrogen [20] A review ofthe CCC method for positron scattering has been givenby Kadyrov and Bray [21]

On the computational side the CCC codes have beenparallelized to use OpenMP on a node and MPI betweennodes and have been deployed successfully on Comet andStampede2 A GPU implementation currently under-way shows immense promise with up to two orders ofmagnitude speedup

C UKRMol(+)

The UK Molecular R-matrix codes were designed totreat low-energy elastic and inelastic electron-moleculecollisions using the R-matrix method they have evolvedto also study photoionization and positron-molecule col-lisions as well as to produce the input required for time-dependent molecular R-matrix with time dependence(RMT) calculations [22] Similar to BSR they are basedon the R-matrix method and the general theory can befound in the book by Burke [9]

The now frozen release version of the (mainly serial)UKRMol suite uses Gaussian Type Orbitals (GTOs) torepresent both the target and continuum orbitals A pub-lication presenting this code by Carr et al [23] followed aproject which substantially updated (to Fortran95) andstandardized the programming used particularly in theinner region A review article by Tennyson [25] from thesame period gives a comprehensive overview of theoryused and the functionality of the code

The use of GTOs to represent the continuum leads toconstraints on both the size of the inner region that canbe used and the free electron energy range Recently anew suite known as UKRMol+ has been developed ledby Zdenek Masın and Jimena Gorfinkiel [23] The codeuses the new GBTOlib integral library to determine allthe one- and two-electron integrals needed in the mixedbasis of GTOs and B-splines it offers the choice of using

4

GTOs B-splines or hybrid GTOs ndash B-splines to representthe continuum the bound orbitals are always describedby GTOs The library which uses object oriented fea-tures from the Fortran2003 standard involves distributedand shared-memory parallelization UKRMol+ incorpo-rates a number of algorithmic improvements including afaster configuration state function (CSF) generation andparallization of the construction and diagonalization ofthe N and (N + 1) Hamiltonians [29] Further paral-lelization to avoid IO to disk during the evaulation oftransition moments for photoionization and RMT inputis currently being tested

Both suites contain a rich array of outer region func-tionality including automated resonance detection andfitting bound state detection computation of multichan-nel quantum defects rotational excitation and evaluationof photoionization cross sections So far applications ofthe UKRMol+ suite are limited [26ndash28] but a full releaseand associated article will be available shortly [29]

The codes have been available as freeware for morethan a decade and are widely used the software canbe downloaded as a tarball and installed (in the case ofUKRmol+) using cmake provided the necessary librariesare available in the system Neither suite is straightfor-ward to use without training Quantemol-N [30] is a com-mercial front end which has led to a significant increase inthe user base of the code A set of perl scripts developedby Karel Houfek that simplify writing the input is alsonow available both for electron scattering and photoion-ization calculations Further details can be found on thewebsite of the UK Atomic Molecular and Optical physicsR-matrix consortium (httpswwwukamorcom)

D tRecX

The tRecX code package [31 32] is a general frameworkfor solving initial value problems of the form

part

parttΨ = D[Ψ t] + Φ(t) (1)

for an arbitrary number of spatial dimensions and a vari-ety of coordinate systems The main design is for linearD but non-linear operators can also be used

1 Applications

The code has been primarily used for solving the time-dependent Schrodinger equation of atomic and molecularsystems in ultra-short pulses and in strong near-IR fieldsThe most significant results are fully differential spectrafor single- and double photo-emission from the He atomat near infrared wave-length [33] strong field ionizationrates of noble gases [34] and differential spectra for smalldi- and tri-atomic molecules [35 36] cf Fig 1

FIG 1 Left Helium in full dimensions double emissioncross-section σ(pz1 pz2) for a 2-cycle pulse at wavelength 400nm and 5times1014 Wcm2 intensity Anti-correlated emission isfavored Right haCC calculation for CO2 photo-emission byan 800 nm laser pulse at intensity 1014 Wcm2 up to energies25 au in the xz-plane at 45 alignment of the molecular axisto polarization direction (from [36])

2 Methods

The code uses three newly developed key techniques

1 irECS mdash ldquoInfinite range exterior complex scal-ingrdquo [37] as absorbing boundary conditions and forthe computation of life-times irECS is a variantof exterior complex scaling with an infinitely wideboundary for absorption at all wave lengths

2 tSurff mdash the time-dependent surface fluxmethod [38] for photo-emission spectra BytSurff the actual numerical solution remainscontained in a small reactive region of typically 20to 100 atomic units

3 haCC mdash the ldquohybrid anti-symmetrized CoupledChannelsrdquo method [39] combines Gaussian-basedneutral and ionic CI states with a numerical single-electron basis The numerical basis extends overthe whole system thus ensuring the proper descrip-tion of energetic electron-molecule collisions

Two of these techniques are reflected in the codersquos nametRecX = tSurff + irECS

3 Structure and inputs

An effort was made to keep the code flexible with-out sacrificing efficiency Systems with dimensions fromone (popular models) to six (He in elliptically polarizedfields) as well as multi-channel models (photo-electronspectra for molecules) are treated within the same frame-work the degrees of freedom map into a tree hierarchyinducing a tree-structure for vectors and operators andproducing transparent and efficient code by recursion

Basis functions are arranged in a tensor-tree with avariety of basis sets finite-elements FE-DVR and gridsthat can be combined on any number of coordinate axeswith multiple basis sets on the same axis Discretizationand operators can all be input-controlled For example

5

define BOX 20Axis namenCoefflowerupperfuncsorderPhi5expImEta3-11 assocLegendrePhiRn60 0BOXpolynomial15Rn20 BOXInftypolExp[05]

with Eta for cos θ defines polar coordinates The bases

exp(imφ) and P|m|l (cos θ) combine to the spherical har-

monics up to l = 2 and the r-coordinate uses 6015=4FE-DVR elements on [0 20] with 15 polynomials on eachand 20 exponentially damped polynomials at the endAn example for operator specification is

Operator hamiltonian05(ltd_1_dgtlt1gt+lt1gtltd_1_dgt+ltQQgtlt1gt+lt1gtltQQgt)

for (larrminuspart xminusrarrpart x +

larrminuspart yminusrarrpart y + x2 + y2)2 Many standard oper-

ators are pre-defined for various coordinate systems

4 Software

The code is open source hosted on a Gitlab reposi-tory [32] It is written mostly in C++ and linked withsome Fortran-based libraries Optionally functionalitycan be extended by linking FFTW and GSL Standardcompilers are gcc and Intel ports to Windows and MacrsquosClang have been successful but are not actively sup-ported Compilation is through CMake and Doxygendocumentation is available Tutorials and further mate-rials are available in the git and from the tRecX web-site [31]

E XChem

XChem [40 41] is a solution for an all-electron ab initiocalculation of the electronic continuum of molecular sys-tems XChem combines the tools of quantum chemistry(as implemented eg in MOLCAS [44]) and scatteringtheory to accurately account for electron correlation inthe single-ionization continuum of atoms [41 43] smalland medium-size molecules [42] At its core lies a close-coupling expansion combined with the use of a hybridGaussian and B-Spline basis set [40]

This approach yields the scattering states of the molec-ular system via the eigenstates of the close-coupling ma-trix (CCM) While useful in their own right the full po-tential lies in using the close-coupling matrix as a startingpoint for time-dependent calculations In doing so onemay explicitly model the interaction of molecules withultrashort (attosecond) pulses The large band widthsof such pulses lead to the coherent excitation of multi-ple ionization channels whose coupling (accurately de-scribed in XChem) gives rise to complex phenomena

An attractive feature of XChem is that the architec-ture of the basis functions (Fig 2) and the use of MOL-CAS allow one to describe the electronic continuum of

FIG 2 Ilustration of the XChem basis architecture inbenzene Cyan B-splines mangenta Gaussians at the cen-ter of mass of the molecule blue and black Gaussians at theatomic sites not overlapping with B-splines

medium-size molecules at the same level of theory asmulti-reference CI methods do for the ground and thelowest excited states of such molecules At present thelargest systems treated have of the order of ten atoms

1 What can XChem do

XChem can compute

bull The CCM for a user-defined set of ionization chan-nels (each defined as an ionized molecular state cou-pled to electrons of given angular momenta) and in-cluding short range states relevant to the problemat hand

bull Scattering states and scattering phases by asymp-totic fitting to the analytical solution

bull Total and partial photoionization cross sectionswithin perturbation theory

bull Lifetime and character of resonances embedded inthe molecular continuum either via analysis of thecross section or via inclusion of a complex absorbingpotential in the CCM yielding complex eigenener-gies

bull The electron dynamics during and after ionizationcaused by and probed with ultrashort laser pulsesby solving the time-dependent Schrodinger equa-tion using the CCM

bull The angular distribution of photo electrons (inprogress)

2 What can XChem be used for

XChem is a valuable tool for

bull The theoretical study of ultrafast processes in inAttosecond pump-probe experiments AttosecondTransient Absorption Spectroscopy (ATAS) and

6

Reconstruction of Attosecond Beatings by Interfer-ence of Two-Photon Transitions (RABBIT)

bull The investigation of photoionization of complexmolecules close to threshold where electron corre-lation effects are crucial to describe the photoion-ization cross sections

bull The study of ionization processes intrinsically de-pendent on electron correlation like autoionizationand Auger decay

bull The computation of potential energy surfaces forthe investigation of molecular dynamics during andafter fast photoionization events

3 Who is using XChem

bull Researchers in (computational) quantum chemistryor molecular physics interested in studying electrondynamics in the ionization continuum of molecules(eg photoionization charge migration etc)

bull Laboratories investigating ultrafast phenomenain many-electron atoms small and medium-sizemolecular systems

F Other Interesting Software

We are also standing up the rather old Many Elec-tron Structure Applications (MESA) code that was de-veloped at Los Alamos in the 1980rsquos and modified tocompute electron scattering and photoionization crosssections from polyatomic molecules using the ComplexKohn Method [45] While this code is old and in need ofsignificant modernization it was built to compute elec-tron scattering and photoionization transition matrix el-ements for general polyatomic molecules a capability notpresent in other codes and of interest to many users

There is also an effort underway to incorporateMOLSCAT [46] a heavy particle collision code forvibrational-rotational scattering in molecules Table II Fsummarizes some of the applications deployed

III AMP SCIENCE GATEWAY DEPLOYMENTAND APPLICATION INTEGRATION

The AMP Gateway is deployed using the Apache Aira-vata Framework [4] It relies on the Science GatewayPlatform as a Service (SciGaP) (httpsscigaporg)hosting services [3] at Indiana University The Sci-GaP platform provides gateway services using the multi-tenanted Apache Airavata middleware The Airavatacore enables features such as managing user identityaccounts authorization provisioning and the ability to

access XSEDE and other high performance computa-tional resources such as XSEDErsquos Stampede2 Comet andBridges These resources are transparently integratedinto the AMP gateway and the batch queues are usedfor scheduling the execution of models using applicationsand user defined parameters

A User Accounts Authentication andAuthorization

The AMP gateway user accounts can be created byusers by providing a userid and setting a password alongwith providing email for verification process In addi-tion to this an automated process using the user insti-tutional login via CI-Logon [47] is also provided whichavoids the email verification step The gateway adminis-trator controls the access to the resources and needs toapprove the user for gateway resource access The usersget a ldquogateway pending rolerdquo when they register Thegateway middleware provides authentication and autho-rization services through the Keycloak [2] identity man-agement system supported by SciGaP Once the gatewayadministrator provides approval the role of the new userwill be changed to ldquogateway-userrdquo which enables accessto gateway resources and applications The gateway ad-ditionally provides ldquoadmin-read-onlyrdquo and ldquoadmin rolesrdquowith their associated permissions for a group to sharethe admin responsibilities and reporting purposes Aldquogateway-userrdquo then can use gateway services such ascreating monitoring sharing and cloning experiments(computational simulations) The users can also addtheir own compute resource allocations using the func-tions available under ldquoUser Settingsrdquo The ldquoadminrdquo usershave the authorization to control metadata for accessingXSEDE through the gateway ldquocommunity loginrdquo regis-ter and deploy applications and their (user) interfacesmanage users and monitor and access all user experi-ments These privileges enable the admin to efficientlytroubleshoot any issues relating to the user services Theldquoadmin-read-onlyrdquo users can view all ldquoadminrdquo related in-formation but not modify any of the settings

B AMP Application Deployment and UserInterface Creation

The AMP gateway started with four specific applica-tions described above BSR XChem CCC and tRecXsuites These applications were compiled and tested on anumber of the XSEDE resources Each of them requiresa different set of libraries and in the case of XChem inte-gration with other Open source software such as Open-Molcas [12] They were independently tested by the col-laborating XSEDE ECSS consultant when deployed bythe developers or deployed by the consultant to estab-lish the required environments and tweak makecmakedata The Application deployment in the AMP Gate-

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

4

GTOs B-splines or hybrid GTOs ndash B-splines to representthe continuum the bound orbitals are always describedby GTOs The library which uses object oriented fea-tures from the Fortran2003 standard involves distributedand shared-memory parallelization UKRMol+ incorpo-rates a number of algorithmic improvements including afaster configuration state function (CSF) generation andparallization of the construction and diagonalization ofthe N and (N + 1) Hamiltonians [29] Further paral-lelization to avoid IO to disk during the evaulation oftransition moments for photoionization and RMT inputis currently being tested

Both suites contain a rich array of outer region func-tionality including automated resonance detection andfitting bound state detection computation of multichan-nel quantum defects rotational excitation and evaluationof photoionization cross sections So far applications ofthe UKRMol+ suite are limited [26ndash28] but a full releaseand associated article will be available shortly [29]

The codes have been available as freeware for morethan a decade and are widely used the software canbe downloaded as a tarball and installed (in the case ofUKRmol+) using cmake provided the necessary librariesare available in the system Neither suite is straightfor-ward to use without training Quantemol-N [30] is a com-mercial front end which has led to a significant increase inthe user base of the code A set of perl scripts developedby Karel Houfek that simplify writing the input is alsonow available both for electron scattering and photoion-ization calculations Further details can be found on thewebsite of the UK Atomic Molecular and Optical physicsR-matrix consortium (httpswwwukamorcom)

D tRecX

The tRecX code package [31 32] is a general frameworkfor solving initial value problems of the form

part

parttΨ = D[Ψ t] + Φ(t) (1)

for an arbitrary number of spatial dimensions and a vari-ety of coordinate systems The main design is for linearD but non-linear operators can also be used

1 Applications

The code has been primarily used for solving the time-dependent Schrodinger equation of atomic and molecularsystems in ultra-short pulses and in strong near-IR fieldsThe most significant results are fully differential spectrafor single- and double photo-emission from the He atomat near infrared wave-length [33] strong field ionizationrates of noble gases [34] and differential spectra for smalldi- and tri-atomic molecules [35 36] cf Fig 1

FIG 1 Left Helium in full dimensions double emissioncross-section σ(pz1 pz2) for a 2-cycle pulse at wavelength 400nm and 5times1014 Wcm2 intensity Anti-correlated emission isfavored Right haCC calculation for CO2 photo-emission byan 800 nm laser pulse at intensity 1014 Wcm2 up to energies25 au in the xz-plane at 45 alignment of the molecular axisto polarization direction (from [36])

2 Methods

The code uses three newly developed key techniques

1 irECS mdash ldquoInfinite range exterior complex scal-ingrdquo [37] as absorbing boundary conditions and forthe computation of life-times irECS is a variantof exterior complex scaling with an infinitely wideboundary for absorption at all wave lengths

2 tSurff mdash the time-dependent surface fluxmethod [38] for photo-emission spectra BytSurff the actual numerical solution remainscontained in a small reactive region of typically 20to 100 atomic units

3 haCC mdash the ldquohybrid anti-symmetrized CoupledChannelsrdquo method [39] combines Gaussian-basedneutral and ionic CI states with a numerical single-electron basis The numerical basis extends overthe whole system thus ensuring the proper descrip-tion of energetic electron-molecule collisions

Two of these techniques are reflected in the codersquos nametRecX = tSurff + irECS

3 Structure and inputs

An effort was made to keep the code flexible with-out sacrificing efficiency Systems with dimensions fromone (popular models) to six (He in elliptically polarizedfields) as well as multi-channel models (photo-electronspectra for molecules) are treated within the same frame-work the degrees of freedom map into a tree hierarchyinducing a tree-structure for vectors and operators andproducing transparent and efficient code by recursion

Basis functions are arranged in a tensor-tree with avariety of basis sets finite-elements FE-DVR and gridsthat can be combined on any number of coordinate axeswith multiple basis sets on the same axis Discretizationand operators can all be input-controlled For example

5

define BOX 20Axis namenCoefflowerupperfuncsorderPhi5expImEta3-11 assocLegendrePhiRn60 0BOXpolynomial15Rn20 BOXInftypolExp[05]

with Eta for cos θ defines polar coordinates The bases

exp(imφ) and P|m|l (cos θ) combine to the spherical har-

monics up to l = 2 and the r-coordinate uses 6015=4FE-DVR elements on [0 20] with 15 polynomials on eachand 20 exponentially damped polynomials at the endAn example for operator specification is

Operator hamiltonian05(ltd_1_dgtlt1gt+lt1gtltd_1_dgt+ltQQgtlt1gt+lt1gtltQQgt)

for (larrminuspart xminusrarrpart x +

larrminuspart yminusrarrpart y + x2 + y2)2 Many standard oper-

ators are pre-defined for various coordinate systems

4 Software

The code is open source hosted on a Gitlab reposi-tory [32] It is written mostly in C++ and linked withsome Fortran-based libraries Optionally functionalitycan be extended by linking FFTW and GSL Standardcompilers are gcc and Intel ports to Windows and MacrsquosClang have been successful but are not actively sup-ported Compilation is through CMake and Doxygendocumentation is available Tutorials and further mate-rials are available in the git and from the tRecX web-site [31]

E XChem

XChem [40 41] is a solution for an all-electron ab initiocalculation of the electronic continuum of molecular sys-tems XChem combines the tools of quantum chemistry(as implemented eg in MOLCAS [44]) and scatteringtheory to accurately account for electron correlation inthe single-ionization continuum of atoms [41 43] smalland medium-size molecules [42] At its core lies a close-coupling expansion combined with the use of a hybridGaussian and B-Spline basis set [40]

This approach yields the scattering states of the molec-ular system via the eigenstates of the close-coupling ma-trix (CCM) While useful in their own right the full po-tential lies in using the close-coupling matrix as a startingpoint for time-dependent calculations In doing so onemay explicitly model the interaction of molecules withultrashort (attosecond) pulses The large band widthsof such pulses lead to the coherent excitation of multi-ple ionization channels whose coupling (accurately de-scribed in XChem) gives rise to complex phenomena

An attractive feature of XChem is that the architec-ture of the basis functions (Fig 2) and the use of MOL-CAS allow one to describe the electronic continuum of

FIG 2 Ilustration of the XChem basis architecture inbenzene Cyan B-splines mangenta Gaussians at the cen-ter of mass of the molecule blue and black Gaussians at theatomic sites not overlapping with B-splines

medium-size molecules at the same level of theory asmulti-reference CI methods do for the ground and thelowest excited states of such molecules At present thelargest systems treated have of the order of ten atoms

1 What can XChem do

XChem can compute

bull The CCM for a user-defined set of ionization chan-nels (each defined as an ionized molecular state cou-pled to electrons of given angular momenta) and in-cluding short range states relevant to the problemat hand

bull Scattering states and scattering phases by asymp-totic fitting to the analytical solution

bull Total and partial photoionization cross sectionswithin perturbation theory

bull Lifetime and character of resonances embedded inthe molecular continuum either via analysis of thecross section or via inclusion of a complex absorbingpotential in the CCM yielding complex eigenener-gies

bull The electron dynamics during and after ionizationcaused by and probed with ultrashort laser pulsesby solving the time-dependent Schrodinger equa-tion using the CCM

bull The angular distribution of photo electrons (inprogress)

2 What can XChem be used for

XChem is a valuable tool for

bull The theoretical study of ultrafast processes in inAttosecond pump-probe experiments AttosecondTransient Absorption Spectroscopy (ATAS) and

6

Reconstruction of Attosecond Beatings by Interfer-ence of Two-Photon Transitions (RABBIT)

bull The investigation of photoionization of complexmolecules close to threshold where electron corre-lation effects are crucial to describe the photoion-ization cross sections

bull The study of ionization processes intrinsically de-pendent on electron correlation like autoionizationand Auger decay

bull The computation of potential energy surfaces forthe investigation of molecular dynamics during andafter fast photoionization events

3 Who is using XChem

bull Researchers in (computational) quantum chemistryor molecular physics interested in studying electrondynamics in the ionization continuum of molecules(eg photoionization charge migration etc)

bull Laboratories investigating ultrafast phenomenain many-electron atoms small and medium-sizemolecular systems

F Other Interesting Software

We are also standing up the rather old Many Elec-tron Structure Applications (MESA) code that was de-veloped at Los Alamos in the 1980rsquos and modified tocompute electron scattering and photoionization crosssections from polyatomic molecules using the ComplexKohn Method [45] While this code is old and in need ofsignificant modernization it was built to compute elec-tron scattering and photoionization transition matrix el-ements for general polyatomic molecules a capability notpresent in other codes and of interest to many users

There is also an effort underway to incorporateMOLSCAT [46] a heavy particle collision code forvibrational-rotational scattering in molecules Table II Fsummarizes some of the applications deployed

III AMP SCIENCE GATEWAY DEPLOYMENTAND APPLICATION INTEGRATION

The AMP Gateway is deployed using the Apache Aira-vata Framework [4] It relies on the Science GatewayPlatform as a Service (SciGaP) (httpsscigaporg)hosting services [3] at Indiana University The Sci-GaP platform provides gateway services using the multi-tenanted Apache Airavata middleware The Airavatacore enables features such as managing user identityaccounts authorization provisioning and the ability to

access XSEDE and other high performance computa-tional resources such as XSEDErsquos Stampede2 Comet andBridges These resources are transparently integratedinto the AMP gateway and the batch queues are usedfor scheduling the execution of models using applicationsand user defined parameters

A User Accounts Authentication andAuthorization

The AMP gateway user accounts can be created byusers by providing a userid and setting a password alongwith providing email for verification process In addi-tion to this an automated process using the user insti-tutional login via CI-Logon [47] is also provided whichavoids the email verification step The gateway adminis-trator controls the access to the resources and needs toapprove the user for gateway resource access The usersget a ldquogateway pending rolerdquo when they register Thegateway middleware provides authentication and autho-rization services through the Keycloak [2] identity man-agement system supported by SciGaP Once the gatewayadministrator provides approval the role of the new userwill be changed to ldquogateway-userrdquo which enables accessto gateway resources and applications The gateway ad-ditionally provides ldquoadmin-read-onlyrdquo and ldquoadmin rolesrdquowith their associated permissions for a group to sharethe admin responsibilities and reporting purposes Aldquogateway-userrdquo then can use gateway services such ascreating monitoring sharing and cloning experiments(computational simulations) The users can also addtheir own compute resource allocations using the func-tions available under ldquoUser Settingsrdquo The ldquoadminrdquo usershave the authorization to control metadata for accessingXSEDE through the gateway ldquocommunity loginrdquo regis-ter and deploy applications and their (user) interfacesmanage users and monitor and access all user experi-ments These privileges enable the admin to efficientlytroubleshoot any issues relating to the user services Theldquoadmin-read-onlyrdquo users can view all ldquoadminrdquo related in-formation but not modify any of the settings

B AMP Application Deployment and UserInterface Creation

The AMP gateway started with four specific applica-tions described above BSR XChem CCC and tRecXsuites These applications were compiled and tested on anumber of the XSEDE resources Each of them requiresa different set of libraries and in the case of XChem inte-gration with other Open source software such as Open-Molcas [12] They were independently tested by the col-laborating XSEDE ECSS consultant when deployed bythe developers or deployed by the consultant to estab-lish the required environments and tweak makecmakedata The Application deployment in the AMP Gate-

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

5

define BOX 20Axis namenCoefflowerupperfuncsorderPhi5expImEta3-11 assocLegendrePhiRn60 0BOXpolynomial15Rn20 BOXInftypolExp[05]

with Eta for cos θ defines polar coordinates The bases

exp(imφ) and P|m|l (cos θ) combine to the spherical har-

monics up to l = 2 and the r-coordinate uses 6015=4FE-DVR elements on [0 20] with 15 polynomials on eachand 20 exponentially damped polynomials at the endAn example for operator specification is

Operator hamiltonian05(ltd_1_dgtlt1gt+lt1gtltd_1_dgt+ltQQgtlt1gt+lt1gtltQQgt)

for (larrminuspart xminusrarrpart x +

larrminuspart yminusrarrpart y + x2 + y2)2 Many standard oper-

ators are pre-defined for various coordinate systems

4 Software

The code is open source hosted on a Gitlab reposi-tory [32] It is written mostly in C++ and linked withsome Fortran-based libraries Optionally functionalitycan be extended by linking FFTW and GSL Standardcompilers are gcc and Intel ports to Windows and MacrsquosClang have been successful but are not actively sup-ported Compilation is through CMake and Doxygendocumentation is available Tutorials and further mate-rials are available in the git and from the tRecX web-site [31]

E XChem

XChem [40 41] is a solution for an all-electron ab initiocalculation of the electronic continuum of molecular sys-tems XChem combines the tools of quantum chemistry(as implemented eg in MOLCAS [44]) and scatteringtheory to accurately account for electron correlation inthe single-ionization continuum of atoms [41 43] smalland medium-size molecules [42] At its core lies a close-coupling expansion combined with the use of a hybridGaussian and B-Spline basis set [40]

This approach yields the scattering states of the molec-ular system via the eigenstates of the close-coupling ma-trix (CCM) While useful in their own right the full po-tential lies in using the close-coupling matrix as a startingpoint for time-dependent calculations In doing so onemay explicitly model the interaction of molecules withultrashort (attosecond) pulses The large band widthsof such pulses lead to the coherent excitation of multi-ple ionization channels whose coupling (accurately de-scribed in XChem) gives rise to complex phenomena

An attractive feature of XChem is that the architec-ture of the basis functions (Fig 2) and the use of MOL-CAS allow one to describe the electronic continuum of

FIG 2 Ilustration of the XChem basis architecture inbenzene Cyan B-splines mangenta Gaussians at the cen-ter of mass of the molecule blue and black Gaussians at theatomic sites not overlapping with B-splines

medium-size molecules at the same level of theory asmulti-reference CI methods do for the ground and thelowest excited states of such molecules At present thelargest systems treated have of the order of ten atoms

1 What can XChem do

XChem can compute

bull The CCM for a user-defined set of ionization chan-nels (each defined as an ionized molecular state cou-pled to electrons of given angular momenta) and in-cluding short range states relevant to the problemat hand

bull Scattering states and scattering phases by asymp-totic fitting to the analytical solution

bull Total and partial photoionization cross sectionswithin perturbation theory

bull Lifetime and character of resonances embedded inthe molecular continuum either via analysis of thecross section or via inclusion of a complex absorbingpotential in the CCM yielding complex eigenener-gies

bull The electron dynamics during and after ionizationcaused by and probed with ultrashort laser pulsesby solving the time-dependent Schrodinger equa-tion using the CCM

bull The angular distribution of photo electrons (inprogress)

2 What can XChem be used for

XChem is a valuable tool for

bull The theoretical study of ultrafast processes in inAttosecond pump-probe experiments AttosecondTransient Absorption Spectroscopy (ATAS) and

6

Reconstruction of Attosecond Beatings by Interfer-ence of Two-Photon Transitions (RABBIT)

bull The investigation of photoionization of complexmolecules close to threshold where electron corre-lation effects are crucial to describe the photoion-ization cross sections

bull The study of ionization processes intrinsically de-pendent on electron correlation like autoionizationand Auger decay

bull The computation of potential energy surfaces forthe investigation of molecular dynamics during andafter fast photoionization events

3 Who is using XChem

bull Researchers in (computational) quantum chemistryor molecular physics interested in studying electrondynamics in the ionization continuum of molecules(eg photoionization charge migration etc)

bull Laboratories investigating ultrafast phenomenain many-electron atoms small and medium-sizemolecular systems

F Other Interesting Software

We are also standing up the rather old Many Elec-tron Structure Applications (MESA) code that was de-veloped at Los Alamos in the 1980rsquos and modified tocompute electron scattering and photoionization crosssections from polyatomic molecules using the ComplexKohn Method [45] While this code is old and in need ofsignificant modernization it was built to compute elec-tron scattering and photoionization transition matrix el-ements for general polyatomic molecules a capability notpresent in other codes and of interest to many users

There is also an effort underway to incorporateMOLSCAT [46] a heavy particle collision code forvibrational-rotational scattering in molecules Table II Fsummarizes some of the applications deployed

III AMP SCIENCE GATEWAY DEPLOYMENTAND APPLICATION INTEGRATION

The AMP Gateway is deployed using the Apache Aira-vata Framework [4] It relies on the Science GatewayPlatform as a Service (SciGaP) (httpsscigaporg)hosting services [3] at Indiana University The Sci-GaP platform provides gateway services using the multi-tenanted Apache Airavata middleware The Airavatacore enables features such as managing user identityaccounts authorization provisioning and the ability to

access XSEDE and other high performance computa-tional resources such as XSEDErsquos Stampede2 Comet andBridges These resources are transparently integratedinto the AMP gateway and the batch queues are usedfor scheduling the execution of models using applicationsand user defined parameters

A User Accounts Authentication andAuthorization

The AMP gateway user accounts can be created byusers by providing a userid and setting a password alongwith providing email for verification process In addi-tion to this an automated process using the user insti-tutional login via CI-Logon [47] is also provided whichavoids the email verification step The gateway adminis-trator controls the access to the resources and needs toapprove the user for gateway resource access The usersget a ldquogateway pending rolerdquo when they register Thegateway middleware provides authentication and autho-rization services through the Keycloak [2] identity man-agement system supported by SciGaP Once the gatewayadministrator provides approval the role of the new userwill be changed to ldquogateway-userrdquo which enables accessto gateway resources and applications The gateway ad-ditionally provides ldquoadmin-read-onlyrdquo and ldquoadmin rolesrdquowith their associated permissions for a group to sharethe admin responsibilities and reporting purposes Aldquogateway-userrdquo then can use gateway services such ascreating monitoring sharing and cloning experiments(computational simulations) The users can also addtheir own compute resource allocations using the func-tions available under ldquoUser Settingsrdquo The ldquoadminrdquo usershave the authorization to control metadata for accessingXSEDE through the gateway ldquocommunity loginrdquo regis-ter and deploy applications and their (user) interfacesmanage users and monitor and access all user experi-ments These privileges enable the admin to efficientlytroubleshoot any issues relating to the user services Theldquoadmin-read-onlyrdquo users can view all ldquoadminrdquo related in-formation but not modify any of the settings

B AMP Application Deployment and UserInterface Creation

The AMP gateway started with four specific applica-tions described above BSR XChem CCC and tRecXsuites These applications were compiled and tested on anumber of the XSEDE resources Each of them requiresa different set of libraries and in the case of XChem inte-gration with other Open source software such as Open-Molcas [12] They were independently tested by the col-laborating XSEDE ECSS consultant when deployed bythe developers or deployed by the consultant to estab-lish the required environments and tweak makecmakedata The Application deployment in the AMP Gate-

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

6

Reconstruction of Attosecond Beatings by Interfer-ence of Two-Photon Transitions (RABBIT)

bull The investigation of photoionization of complexmolecules close to threshold where electron corre-lation effects are crucial to describe the photoion-ization cross sections

bull The study of ionization processes intrinsically de-pendent on electron correlation like autoionizationand Auger decay

bull The computation of potential energy surfaces forthe investigation of molecular dynamics during andafter fast photoionization events

3 Who is using XChem

bull Researchers in (computational) quantum chemistryor molecular physics interested in studying electrondynamics in the ionization continuum of molecules(eg photoionization charge migration etc)

bull Laboratories investigating ultrafast phenomenain many-electron atoms small and medium-sizemolecular systems

F Other Interesting Software

We are also standing up the rather old Many Elec-tron Structure Applications (MESA) code that was de-veloped at Los Alamos in the 1980rsquos and modified tocompute electron scattering and photoionization crosssections from polyatomic molecules using the ComplexKohn Method [45] While this code is old and in need ofsignificant modernization it was built to compute elec-tron scattering and photoionization transition matrix el-ements for general polyatomic molecules a capability notpresent in other codes and of interest to many users

There is also an effort underway to incorporateMOLSCAT [46] a heavy particle collision code forvibrational-rotational scattering in molecules Table II Fsummarizes some of the applications deployed

III AMP SCIENCE GATEWAY DEPLOYMENTAND APPLICATION INTEGRATION

The AMP Gateway is deployed using the Apache Aira-vata Framework [4] It relies on the Science GatewayPlatform as a Service (SciGaP) (httpsscigaporg)hosting services [3] at Indiana University The Sci-GaP platform provides gateway services using the multi-tenanted Apache Airavata middleware The Airavatacore enables features such as managing user identityaccounts authorization provisioning and the ability to

access XSEDE and other high performance computa-tional resources such as XSEDErsquos Stampede2 Comet andBridges These resources are transparently integratedinto the AMP gateway and the batch queues are usedfor scheduling the execution of models using applicationsand user defined parameters

A User Accounts Authentication andAuthorization

The AMP gateway user accounts can be created byusers by providing a userid and setting a password alongwith providing email for verification process In addi-tion to this an automated process using the user insti-tutional login via CI-Logon [47] is also provided whichavoids the email verification step The gateway adminis-trator controls the access to the resources and needs toapprove the user for gateway resource access The usersget a ldquogateway pending rolerdquo when they register Thegateway middleware provides authentication and autho-rization services through the Keycloak [2] identity man-agement system supported by SciGaP Once the gatewayadministrator provides approval the role of the new userwill be changed to ldquogateway-userrdquo which enables accessto gateway resources and applications The gateway ad-ditionally provides ldquoadmin-read-onlyrdquo and ldquoadmin rolesrdquowith their associated permissions for a group to sharethe admin responsibilities and reporting purposes Aldquogateway-userrdquo then can use gateway services such ascreating monitoring sharing and cloning experiments(computational simulations) The users can also addtheir own compute resource allocations using the func-tions available under ldquoUser Settingsrdquo The ldquoadminrdquo usershave the authorization to control metadata for accessingXSEDE through the gateway ldquocommunity loginrdquo regis-ter and deploy applications and their (user) interfacesmanage users and monitor and access all user experi-ments These privileges enable the admin to efficientlytroubleshoot any issues relating to the user services Theldquoadmin-read-onlyrdquo users can view all ldquoadminrdquo related in-formation but not modify any of the settings

B AMP Application Deployment and UserInterface Creation

The AMP gateway started with four specific applica-tions described above BSR XChem CCC and tRecXsuites These applications were compiled and tested on anumber of the XSEDE resources Each of them requiresa different set of libraries and in the case of XChem inte-gration with other Open source software such as Open-Molcas [12] They were independently tested by the col-laborating XSEDE ECSS consultant when deployed bythe developers or deployed by the consultant to estab-lish the required environments and tweak makecmakedata The Application deployment in the AMP Gate-

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

7

Code Application Method Parallel Access Restrictions

or Serial

BSR Electron scattering R-matrixB-spline S MPI BSR (serial) atoms atomic ions

Photoionization

Structure

CCC Electron scattering Close-coupling OpenMP Quasi one- and two-

Positron scattering Fredholm equations and MPI electron atoms

Photoionization in momentum space and ions

UKRMol(+) Electron scattering R-matrix Close-coupling OpenMP Public Molecules and

Photoionization Gaussian and B-spline basis and MPI (Zenodo) clusters (le 30 atoms)

tRecX Strong-field photo- TDSE (tSurff haCC) grids MPI Public Small molecules

emission spectra amp rates CI-states FE-DVR bases (Git) two-electron atoms

XChem Scattering states Close-coupling OpenMP Upon request Small and

Photoionization Configuration Interaction medium-size

Hybrid Gaussian and B-spline basis molecules

MESA Electronic structure SCF MCSCF S By request Small to

Electron scattering CI Complex Kohn medium-size

Photoionization molecules

TABLE I Some characteristics of the software suites deployed in the AMP Gateway

way consists of defining the application as a ldquomodulerdquoan ldquointerfacerdquo is defined to user interaction with the ap-plication and a resource specific ldquodeploymentrdquo descrip-tion to fully define it on the gateway The user inter-faces are tailored to each application and enable the usersto provide input parameters using files or variables thateither can be sent as arguments to the application ora wrapping script Currently the interface generatorthe PHP Airavata client API provides ways to definefile URIs variables (strings realintegerBooleans) forinputs and standard ErrorOutputs and file and vari-ables for outputs Multiple choices for different appli-cation sub-modules can be provided to specify and in-voke a specific component of the application(for exampleBoundPhotonicsLSJK under software BSR) This canbe defined using a simple comma delimited set and canbe used in a wrapper to pass it as arguments or speci-fied as input parameters for the application as depictedin Fig 3 The interfaces deployed for the four appli-cations will be further enhanced with additional detailsfor input variables and ingesting file sets as archives andwhole folders in due course The job submission interfacethen enables users to define HPC resource details suchas the systemmachine ID that is automatically sortedfrom the list obtained from the deployment descriptionqueuepartition and allocation registered by the admin-istrator or a user and job specific details such as thenumber of nodes and processors time and memory speci-fications In additions these experiment specification theApache Airavata framework provides a sharing mecha-nism for jobsexperiments and projects (which are col-lections of experiments) to be shared with collaboratorsand can be set during the experiment creation (or at any

time after) This allows collaborative job submissionmonitoring and analysis of the results

FIG 3 Input interface for BSR software to select a specificmodule for execution and provide inputs as a tarball and setthe resource requirements

8

FIG 4 Experiment summary for an job with the batchscript created by Airavata middleware

C Monitoring Job Progress

During experiment creation users can provide theiremail to receive messages at job start and end supplied bythe scheduler Additionally once the experiment is ac-cepted and launched an ldquoExperiment Summaryrdquo inter-face is launched and automatically refreshed periodicallyin order to show the status of the job submitted into theXSEDE resources Currently the status of the job in thescheduler is reflected in the summary interface shown inFig 4 Gateway users can monitor experiments ownedby them or shared with them by other gateway usersGateway administrators can monitor all gateway exper-iments using the ldquoExperiment Statisticsrdquo page availablein the ldquoAdmin Dashboardrdquo The monitoring informa-tion is provided by a log processing system that extractsthe relevant experiment task level execution logs fromthe gateway middleware and presents it in the gatewaymonitoring interfaces as depicted in Fig 5

FIG 5 Admin interface showing task level log for an exper-iment

D Gateway Admin Dashboard

The ldquoAdmin Dashboardrdquo is the workspace for the gate-way administrator(s) within the gateway All the adminfeatures discussed above are available through the AdminDashboard Apart from what has been discussed the ad-min dashboard provides a notification feature and also away to managing gateway preferences when it comes toindividual compute resource and storage resource con-nectivity and also helps managing credentials for securecompute resource communications Gateway administra-tors can create notifications for gateway users and sharethem with the users with set begin and expiration times

9

The gateway admin can set and define preferences for theusage of each compute resource with specifications suchas community or shared login name job scratch locationfor the job data staging and execution preferred job sub-mission and file transfer protocols and allocation projectnumber for charging the run time Similarly the admindashboard is used to generate an SSH credential tokenand key to be used for authentication and authorizationat the compute resource and storage resource communi-cations The admin dashboard also provides job scriptscreated by the Airavata middleware the actual path ofthe job directory on the remote HPC system and detailedtask level logs for a specific job to the administrator tocheck health of the job workflow or troubleshoot in caseit is needed

IV COMMUNITY BUILDING

A proposal has been written to the MOLecular Soft-ware Sciences Institute(MOLSSI) [48] to host a series ofworkshops designed to

1 Promote our ideas to a larger more diverse group ofscientists than the ITAMP workshop participantsboth in AampM physics as well as other related fieldsto help us solidify our ideas

2 Initially conduct a three day workshop most likelyhosted by NIST sometime this fall

bull We envision inviting about 30 participantsconsisting of both AampM scientists and quan-tum chemists

bull We have requested support from the MOLSSIfor the participants and have already been in-formed that we can expect $15K to supportour efforts

bull The NSF Computational Physics program hasalso promised $10K in support

3 The workshop will consist of a number of generalsessions discussing the codes and what they canand cannot do In addition there will be handson sessions on how to use the codes on the ScienceGateway

4 Develop a road-map for other workshops focusingmore on a specific code or codes for specialists

V ACKNOWLEDGEMENTS

BIS acknowledges the Mathematical Software groupof the Applied and Computational Mathematics Divi-sion at NIST for supporting this work This work usedthe Extreme Science and Engineering Discovery Envi-ronment (XSEDE) Stampede2 Comet and Bridges re-sources and an ECSS collaboration grant the throughthe allocation TG-PHY180023 which is supported byNational Science Foundation grant number ACI-1548562Additional support is provided by the Science GatewaysCommunity Institute NSF Award 1547611 [49] The Sci-GaPorg platform and Apache Airavata are supported byNSF Award 1339774 KB and OZ acknowledge NSF sup-port through grant Nos PHY-1520970 PHY-1803844and OAC-1834740 FM and MK acknowledge support bythe ERC Proof-of-Concept Grant No 780284-ImagingXCHEM within the Horizon 2020 Framework Programand by the Spanish MINECO Grant No FIS2016-77889R JT and JDG acknowledge support through theUK-AMOR high end computing consortium under EP-SRC grant EPR0293421

[1] J Towns T Cockerill M Dahan I Foster K GaitherA Grimshaw V Hazlewood S Lathrop D Lifka G DPeterson R Roskies J R Scott and N Wilkins-Diehr2014 Xsede Accelerating scientific discovery CiSE 6 pp6274

[2] M A Christie A Bhandar S Nakandala S Marru EAbeysinghe S Pamidighantam and M E Pierce 2017Using Keycloak for Gateway Authentication and Autho-rization

[3] M E Pierce S Marru E Abeysinghe S Pamidighan-tam M Christie and D Wannipurage 2018 July Sup-porting Science Gateways Using Apache Airavata andSciGaP Services ACM Science Gateways as a PlatformProceedings of the Practice and Experience on AdvancedResearch Computing ( p 99) (accessed 1302019)

[4] M E Pierce S Marru E Abeysinghe S Pamidighan-tam M Christie D Wannipurage Supporting ScienceGateways Using Apache Airavata and SciGaP Services

In Proceedings of the Practice and Experience on Ad-vanced Research Computing ACM Pittsburgh PAUSA 2018 pp 1-4

[5] ITAMP the Institute for Theoretical Atomic and Molecu-lar Physics is an NSF supported institute at the Harvard-Smithsonian which has entered its third decade of ex-istence ITAMP as the leading center of AMO physicstheory in the US has developed a notable reputation fortraining mentoring and sponsoring postdoctoral and vis-iting fellows in theoretical AMO science holding timelyworkshops and recently a Winter Graduate SchoolITAMP facilitates closer interactions between AMO the-ory and experiment and has catalyzed cross-disciplinaryexchanges between AMO and condensed matter physics

[6] O Zatsarinny BSR B-spline atomic R-matrix codesComp Phys Commun 174 (2006) 273

[7] K A Berrington W B Eissner and P H Norring-ton RMATRX1 Belfast atomic R-matrix codes Comp

9

The gateway admin can set and define preferences for theusage of each compute resource with specifications suchas community or shared login name job scratch locationfor the job data staging and execution preferred job sub-mission and file transfer protocols and allocation projectnumber for charging the run time Similarly the admindashboard is used to generate an SSH credential tokenand key to be used for authentication and authorizationat the compute resource and storage resource communi-cations The admin dashboard also provides job scriptscreated by the Airavata middleware the actual path ofthe job directory on the remote HPC system and detailedtask level logs for a specific job to the administrator tocheck health of the job workflow or troubleshoot in caseit is needed

IV COMMUNITY BUILDING

A proposal has been written to the MOLecular Soft-ware Sciences Institute(MOLSSI) [48] to host a series ofworkshops designed to

1 Promote our ideas to a larger more diverse group ofscientists than the ITAMP workshop participantsboth in AampM physics as well as other related fieldsto help us solidify our ideas

2 Initially conduct a three day workshop most likelyhosted by NIST sometime this fall

bull We envision inviting about 30 participantsconsisting of both AampM scientists and quan-tum chemists

bull We have requested support from the MOLSSIfor the participants and have already been in-formed that we can expect $15K to supportour efforts

bull The NSF Computational Physics program hasalso promised $10K in support

3 The workshop will consist of a number of generalsessions discussing the codes and what they canand cannot do In addition there will be handson sessions on how to use the codes on the ScienceGateway

4 Develop a road-map for other workshops focusingmore on a specific code or codes for specialists

V ACKNOWLEDGEMENTS

BIS acknowledges the Mathematical Software groupof the Applied and Computational Mathematics Divi-sion at NIST for supporting this work This work usedthe Extreme Science and Engineering Discovery Envi-ronment (XSEDE) Stampede2 Comet and Bridges re-sources and an ECSS collaboration grant the throughthe allocation TG-PHY180023 which is supported byNational Science Foundation grant number ACI-1548562Additional support is provided by the Science GatewaysCommunity Institute NSF Award 1547611 [49] The Sci-GaPorg platform and Apache Airavata are supported byNSF Award 1339774 KB and OZ acknowledge NSF sup-port through grant Nos PHY-1520970 PHY-1803844and OAC-1834740 FM and MK acknowledge support bythe ERC Proof-of-Concept Grant No 780284-ImagingXCHEM within the Horizon 2020 Framework Programand by the Spanish MINECO Grant No FIS2016-77889R JT and JDG acknowledge support through theUK-AMOR high end computing consortium under EP-SRC grant EPR0293421

[1] J Towns T Cockerill M Dahan I Foster K GaitherA Grimshaw V Hazlewood S Lathrop D Lifka G DPeterson R Roskies J R Scott and N Wilkins-Diehr2014 Xsede Accelerating scientific discovery CiSE 6 pp6274

[2] M A Christie A Bhandar S Nakandala S Marru EAbeysinghe S Pamidighantam and M E Pierce 2017Using Keycloak for Gateway Authentication and Autho-rization

[3] M E Pierce S Marru E Abeysinghe S Pamidighan-tam M Christie and D Wannipurage 2018 July Sup-porting Science Gateways Using Apache Airavata andSciGaP Services ACM Science Gateways as a PlatformProceedings of the Practice and Experience on AdvancedResearch Computing ( p 99) (accessed 1302019)

[4] M E Pierce S Marru E Abeysinghe S Pamidighan-tam M Christie D Wannipurage Supporting ScienceGateways Using Apache Airavata and SciGaP Services

In Proceedings of the Practice and Experience on Ad-vanced Research Computing ACM Pittsburgh PAUSA 2018 pp 1-4

[5] ITAMP the Institute for Theoretical Atomic and Molecu-lar Physics is an NSF supported institute at the Harvard-Smithsonian which has entered its third decade of ex-istence ITAMP as the leading center of AMO physicstheory in the US has developed a notable reputation fortraining mentoring and sponsoring postdoctoral and vis-iting fellows in theoretical AMO science holding timelyworkshops and recently a Winter Graduate SchoolITAMP facilitates closer interactions between AMO the-ory and experiment and has catalyzed cross-disciplinaryexchanges between AMO and condensed matter physics

[6] O Zatsarinny BSR B-spline atomic R-matrix codesComp Phys Commun 174 (2006) 273

[7] K A Berrington W B Eissner and P H Norring-ton RMATRX1 Belfast atomic R-matrix codes Comp

10

Phys Commun 92 (1995) 290[8] N R Badnell httpamdppphysstrathacukrmatrix[9] P G Burke R-Matrix Theory of Atomic Collisions

Springer-Verlag (2011)[10] O Zatsarinny and K Bartschat The B-spline R-matrix

method for atomic processes application to atomicstructure electron collisions and photoionization J PhysB 46 (2013) 112001

[11] Time-dependent recursive indexing Software tRecx[12] OpenMolcas is an open source quantum chemistry code

based on multiconfiguration metyhods such as CASSCFand CASPT2

[13] XCHEM is a code that is designed to compute the inter-action of atoms and molecules with strong short pulseelectromagnetic fields

[14] I Bray and A T Stelbovics Adv Atom Mol Phys 35209 (1995)

[15] I Bray and A T Stelbovics Phys Rev A 46 6995(1992)

[16] I Bray Phys Rev A 49 1066 (1994)[17] D V Fursa and I Bray Phys Rev A 52 1279 (1995)[18] D V Fursa and I Bray J Phys B At Mol Opt Phys

30 5895 (1997)[19] A S Kadyrov and I Bray Phys Rev A 66 012710

(2002)[20] A S Kadyrov C M Rawlins A T Stelbovics I Bray

and M Charlton Phys Rev Lett 114 183201 (2015)[21] A S Kadyrov and I Bray J Phys B At Mol Opt

Phys 49 222002 (2016)[22] Moore L R et al 2011 J Mod Optics 58 1132[23] J M Carr and P G Galiatsatos and J D Gorfinkiel

A G Harvey M A Lysaght D Madden Z Masın MPlummer and J Tennyson Eur Phys J D 66 58 (2012)

[24] A F Al-Refaie and J Tennyson Comput Phys Com-mun 214 216 (2017)

[25] J Tennyson Phys Rep 491 29 (2010)[26] D Darby-Lewis Z Masın and J Tennyson J Phys B

50 17501 (2017)[27] DS Brambila AG Harvey K Houfek Z Masın and O

Smirnova Phys Chem Chem Phys (2017) 19 19673-19682

[28] A Loupas and J D Gorfinkiel J Chem Phys 150064307 (2019)

[29] Z Masın J Benda A G Harvey J D Gorfinkiel andJ Tennyson Comput Phys Comm in preparation

[30] J Tennyson D B Brown J Munro I Rozum H NVarambhia and N Vinci J Phys Conf Ser 86 012001(2007)

[31] ldquoThe tRecX Homepagerdquo ()[32] ldquoThe tRecX git repositoryrdquo ()[33] A Zielinski V P Majety and A Scrinzi Phys Rev A

93 023406 (2016)[34] V P Majety and A Scrinzi Journal of Physics B

Atomic Molecular and Optical Physics 48 245603(2015)

[35] V P Majety and A Scrinzi Phys Rev Lett 115103002 (2015)

[36] V P Majety and A Scrinzi Phys Rev A 96 053421(2017)

[37] A Scrinzi Phys Rev A 81 053845 (2010)[38] A Scrinzi New Journal of Physics 14 085008 (2012)

[39] V P Majety A Zielinski and A Scrinzi New J Phys

17 (2015) 1010881367-2630176063002[40] C Marante L Argenti and F Martın Phys Rev A 90

012506 (2014)[41] C Marante M Klinker I Corral J Gonzalez-Vazquez

L Argenti and F Martın J Chem Theory Comp 13499 (2017)

[42] M Klinker C Marante L Argenti J Gonzalez-Vazquez and F Martın J Phys Chem Lett 9 756(2018)

[43] C Marante M Klinker T Kjellsson E Lindroth JGonzalez-Vazquez L Argenti and F Martın Phys RevA 96 022507 (2017)

[44] F Aquilante J Autschbach R K Carlson L F Chib-otaru M G Delcey L De Vico N Ferre L M FrutosL Gagliardi M Garavelli et al J Comp Chem 37 506(2016)

[45] T N Rescigno C W McCurdy A E Orel and B HLengsfield III The Complex Kohn Variational Methodin ldquoComputational Methods for Electron-Molecule Col-lisionsrdquo 1 (1995)

[46] MOLSCAT is a program for computing non-reactivequantum scattering calculations involving atomic andmolecular particles See Jeremy M Hutson C Ruth LeSueur

[47] CILogon is an open source standards-based service pro-viding the NSF research community with credentials forsecure access to cyberinfrastructure (CI)

[48] Molecular Sciences Software Institute is an organizationlocated in Blacksburg VA which serves as a nexus forscience education and cooperation serving the world-wide community of computational molecular scientists ndasha broad field including of biomolecular simulation quan-tum chemistry and materials science

[49] Pierce ME Marru S Gunathilake L WijeratneDK Singh R Wimalasena C Ratnayaka S andPamidighantam S 2015 Apache Airavata design anddirections of a science gateway framework Concurrencyand Computation Practice and Experience 27(16) pp4282-4291