Download - Planck Simulations
INFSO-RI-508833
Enabling Grids for E-sciencE
www.eu-egee.org
Planck Simulations
Status of the ApplicationC. Vuerli, G. Taffoni, A. Barisani, A. Zacchei, F. Pasian
INAF – Information Systems Unit and OA Trieste
Geneve, 1 March 2006
Wednesday 1 March 2006 2
Enabling Grids for E-sciencE
INFSO-RI-508833
Outline
• Description of the Application and Scientific goals• The Grid added value• Experiences and results with using the EGEE
infrastructure• Future perspectives and issues for use of Grid
Technology• Summing up status
Wednesday 1 March 2006 3
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionThe Planck Mission
• Measure cosmic microwave background (CMB)– succeeds COBE, Boomerang & WMAP missions– aims at even higher resolution
• Timeline– launch August 2007– start of observations 2008– duration >1 year
• Characteristics– continuous data stream (TOD)– large datasets
(a TOD of ~7 TB for the whole LFI mission)– changing calibration (parameters configuration)– high-performance computing for data analysis
Wednesday 1 March 2006 4
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionCOBE & Planck
Wednesday 1 March 2006 5
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionBrief introduction
• Goal: make possible N simulations of the whole Planck/LFI mission (~14 months), each time with different cosmological and instrumental parameters
• Full sky maps production for frequencies 30-857 GHz by means of two complete sky surveys
• Sensitivity of a few μK per pixel 0,3° in amplitude• 22 channels for LFI, 48 for HFI• Data volume produced at the end of the mission: ~2TB
for LFI and ~15TB for HFI• Computing requirements: ~100Tflops for raw data
reduction, foregrounds extraction and CMB maps creation
Wednesday 1 March 2006 6
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionThe Level-S
• Purpose of the Level-S:– ground checks (pre-launch phases);– DPCs Pipelines tuning;– control check & corrections (operational phase).
• Pipeline chained but not parallel(43 executables and a few libraries)
• Languages used are C/C++/Fortran/F90;• Shell/Perl for scripts;• Typical application that benefits by distributed
computing techniques• Porting of Monte Carlo simulation code by Sam Leach
Planck simulation is a set of 70 instances of the Pipeline (22 for LFI and 48 for HFI)
Wednesday 1 March 2006 7
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionThe Level-S
CMB Power Spectrum(cmbfast)
CMB mapsDataanalysis
CMB Map(synfast)
Foregrounds andBeam Patterns
Instrumental Noise Scanning Strategy
TOD
Missionsimulation
Wednesday 1 March 2006 8
Enabling Grids for E-sciencE
INFSO-RI-508833
DescriptionOur application in summary
• Is it parallel? NO it runs concurrently.
• Do we need MPI/parallel? Yes. In later phase for data analysis (16/32 CPUs in the site).
• Do we produce data? YES, we have an intensive data production. Can be more than 1 PB.
• How long does it run? From 6h (short) up to 36h (long)
• Access to/exchange of data coming from other experiments (IVOA, MAGIC)
Wednesday 1 March 2006 9
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid added valuesCPUs and Data
• CPU power:– E-computing lab;– Production burst;– Efficient CPU usage/sharing.
• Data storing/sharing:– Distributed data for distributed users;– Replica and security;– Common interface to software and
data.
Planck simulations are highly computing demanding and produce a huge amount of data. Such resources cannot be usually afforded by a single research institute, both in terms of computing power and data storage space.
Wednesday 1 March 2006 10
Enabling Grids for E-sciencE
INFSO-RI-508833
Grid added valuesQualitative view
• Native collaboration tool;• Common Interface to the users;• Flexible environment;• New approach to data and S/W sharing;• Collaborative work for simulations and reduction:
– less time, less space, less frustration….
• VObs view:– Sharing data over a shared environment;
• Native authentication/authorization mechanism;• A federation of users within a VO fosters the scientific
collaboration;• Collaborative work between different applications.
Wednesday 1 March 2006 11
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsFirst Tests
• First tests performed on a workstation aimed at identifying the computational and storage needs of the simulation SW in detail
LFI 30GHz LFI 44GHz LFI 70GHz
short long short long short long
12 m 389 m 13 m 623 m 17 m 834 m
0.9 GB 34.2 GB 1.2 GB 45.3 GB 1.7 GB 75 GB
Total
long short
255.7 h 1.3 TB 5.5 h 31 GB
Computational time on a dual CPU 2.4 GHz workstation with 2 GB of RAM
for the whole simulation of the LFI mission[4 radiometers at 30 GHz, 6 at 44 GHz and 12 a 70 GHz]
Wednesday 1 March 2006 12
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsGrid Environment
• To allow users to run Planck simulation SW we need to create some specific services on Grid general environment;
• They must be used to run both one or more SDPs (Single Detector Pipeline) or the whole MSJ (Mission Simulation Job);
• They are modular and easy to integrate with new pipeline stages when some upgrade is needed (this is necessary if we want to allow users to develop new codes).
Wednesday 1 March 2006 13
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsGrid Environment
• Step 1: Deployment of the simulations code on the Grid as RPMs. In this first test however we used the Replica Manager to copy and register the SW.
• Step 2: Creation of an application specific environment on top of the UI: a set of Perl scripts are available allowing a user to configure a pipeline and submit it to the Grid.
• Step 3: Implementation of a metadata description to identify the cosmological and instrumental parameters and to associate them to the GUID of a complex output file (TODs, maps, noise contributions etc.) Important for post processing analysis.
Wednesday 1 March 2006 14
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsGrid Environment
Wednesday 1 March 2006 15
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsSimulations description
• A number of test simulations of the MSJ run with the same parameters used on the dual-CPU WS;
• Initially selected only the Grid sites equipped with Xeon WNs at CPU speed of ~2400 and sites with at least 1 free CPU;
• Run sets of MSJ with different degree of parallelization;• Tests repeated 30 times under different load conditions of the Grid
to verify the stability of both the submission tools and of the Grid environment.
• We noticed that RB usually assigns each SDP to a different site, so the MSJ runs on a truly distributed environment. However, a few times the jobs were assigned to the same site but to different WNs. As expected, no significant benefit or decay in performance was noticed in those cases. Also different Grid load did not change significantly the results.
Wednesday 1 March 2006 16
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsSimulations description
• Set of tests involving the whole computing power and data storage available to our VO (~5000 CPUs of different kind). Submission of 100 concurrent MSJs from different UIs with the only requirement of finding enough free disk space to save the output.
• Tests span two years starting from summer 2004 and using different versions of the MW (2.2, 2.4 and 2.6) and within different VOs
• The whole test lasts for ~3 days and was repeated different times under different load conditions of the Grid with no significant change in the results
• Long simulations could require to modify CFTSIO to allow I/O directly on Grid through GridFTP
Wednesday 1 March 2006 17
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsSimulations workflow/1
Wednesday 1 March 2006 18
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsSimulations workflow/2
Wednesday 1 March 2006 19
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsScalability on Grid
0
100
200
300
400
500
600
700
1 2 4 8 16 22
Number of Worker Nodes
Tim
e [
m]
StartTime
SaveTime
ExecTime
Wednesday 1 March 2006 20
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsPost-processing example
• To verify the possibility of using the stored TODs for some kind of post-processing we applied the destriping algorithm on the TODs produced during the short runs.– Metadata are used to locate the files and to identify their GUIDs.– The configuration/submission Planck tools are modified to create
a JDL with the input-file option that points to the TOD GUID.– The input-file option is used by the RB to force the job to run in
the Grid site where the input data are stored. This optimizes the data transfer which is in this way restricted to the site LAN.
– The Grid "configurator" is modified to download any input data file specified in the input-file option before running the pipeline.
Wednesday 1 March 2006 21
Enabling Grids for E-sciencE
INFSO-RI-508833
Experiences and ResultsPost-processing example
• The “destriping” procedure runs for ~20 minutes for 30 GHz channel up to ~40 minutes for the 70 GHz channel on a dual AMD workstation.
• On Grid the run time for a simulations set of 22 radiometer is ~55 minutes with a gain of a factor of 10 in performances compared with times required on the workstation.
Wednesday 1 March 2006 22
Enabling Grids for E-sciencE
INFSO-RI-508833
observed sky after de-striping
map of de-striping residuals
param
maps maps
maps, TOD maps, TOD
param
Node 1
Node k
User Node
CE CE
WNWN ...
Experiences and ResultsPost-processing example
Workstation Grid Gain
short 330 m 25 m 13
long 15342 m 955 m 16
Dual CPU WS 2,4 GHz with 2 GB di RAM vs. Grid
Wednesday 1 March 2006 23
Enabling Grids for E-sciencE
INFSO-RI-508833
Future Perspectives and IssuesProC and G-DSE
• ProC is a scientific workflow engine developed in the framework of IDIS (Integrated Data and Information System) collaboration
It executes “pipelines” of modules• Workflows, directed “acyclic” graphs
It allows the assembly of Pipelines using building block modules• Modules may be heterogeneous (FORTRAN, C, C++, Java, Python,
GDL/IDL, ...); also sub-pipelines It is a data-driven, forward-chaining system
– It has components for ... graphical editing of workflow layouts checking for consistency & completeness Execution
• The G-DSE makes of databases a new embedded resource of the Grid. It enables a new Grid QE to access databases and use them for data analysis (see presentation by G. Taffoni on Thu March 2nd, 2006, at 2.00 PM “Data Access on the Grid” session)
Wednesday 1 March 2006 24
Enabling Grids for E-sciencE
INFSO-RI-508833
Future Perspectives and IssuesPlanck VO Evolution
• Users who joined the VO 15-30 members; • UI in each site;• Quantum-Grid in each site;
Regional Area Current status Future status
R.A.: Italy 15 CPUS 300 GB 15 + 45 CPUS 1 TB (total). All INAF sites are now in the startup phase
R.A.: Spain 30 CPUS 240 GB more
R.A.: France none 6 CPUS 360 GB (total)
R.A.: UK none 2 CPUS 240 GB (total)
R.A.: Germany none 2 CPUS 240 GB (total)
R.A.: The Netherlands none 2 CPUS 240 GB (total)
• VO Planck may currently rely on ~5000 available CPUs
Wednesday 1 March 2006 25
Enabling Grids for E-sciencE
INFSO-RI-508833
Future Perspectives and Issues Open issues
• Technical and administrative management of the VO OK• Basic gridification of the Application OK• Main issues met in 2005
– Slow startup process of Planck VO Slow start up of interactions between Planck VO site managers and
national ROCs Some technical initial problems (e.g. VOMS) The management of the VO has proved to be more complex with respect
our expectations Heterogeneous VO
– Some problem on WN environment– Metadata = DSE (work in progress)– Grid-FS complicated and not user-friendly– Debugging
Wednesday 1 March 2006 26
Enabling Grids for E-sciencE
INFSO-RI-508833
Future Perspectives and Issues Effects/corrective actions
• …therefore:– Still missing shared resources within the VO– The gridification of Planck pipelines has still to be completed– Until now extensive tests involving all nodes of the VO were not
possible
• Corrective action– On-site (at VO sites) meetings and training events (involving
Planck and INAF VOs) addressed to site managers and users and scheduled for the next months for a: Fast startup of VO nodes with new shared resources available; Gridification of new pipelines; Extensive tests within the VO.
• Future strictly dependent from a number of factors: gLite (!?!), support (?), EGEE-2 (?)
Wednesday 1 March 2006 27
Enabling Grids for E-sciencE
INFSO-RI-508833
Summing up status
• VO Setup: Management; Technical Management;
VO manager; Site managers;
RLS; VOMS; RB Planck users cert; Planck sites setup; EGEE site support.
• Application Setup: Basic Gridification; First tests;
INFN production Grid; Benchmarks;
Extended gridification; Data&metadata (GDSE!!!); ProC & DMC gridification; Tests:
Runs; Data;
Wednesday 1 March 2006 28
Enabling Grids for E-sciencE
INFSO-RI-508833
End of Presentation
End of Presentation
Thank you for your attention