introduction to grid technologies and practical usage of...
TRANSCRIPT
![Page 1: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/1.jpg)
Introduction to grid technologies and practical usage of gLite middleware
Pavol Štefko, Lukáš Chlad, Lukáš Fajt,Ondřej Ticháček, Alina Pranovich
![Page 2: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/2.jpg)
Grid Computing
● Grid computing is the collection of computer resources from multiple locations to reach a common goal.
● A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive and inexpensive access to high-end computational capabilities.
● Grid tends to be more loosely coupled, heterogenous and geographically dispersed than cluster computing.
● Virtual organization is a set of individuals and/or institutions.
![Page 3: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/3.jpg)
Large Hadron Collider
![Page 4: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/4.jpg)
Large Hadron Collider
● Enormous data stream - 15 PB per year.● More than 1000 active participants.● Necessity for a joint access to the experimental data in the global
scale and their processing and analysis on the computing resources wordwide.
● development of the new technology of distributed computing = grid ● Worldwide LHC Computing Grid (WLCG) : Tier-0 (CERN), Tier-1
(11), Tier-2 (JINR)● Tier-3 Monitoring toolkit, DDM DQ2 Deletion service, Remote
access to ATLAS and CMS,
![Page 5: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/5.jpg)
Prime Grid
● Distributed computing project for searching for PN of world-record size.
● As of September 2011, there was about 7500 active participants from 114 countries with 1.663 petaflops (quadrillion operations per second) of processing power
● Sub-projects○ 321 Prime Search○ Twin Prime Search○ AP26 Search
● Also you can join to the one of the Prime Grid projects● With a little patience, you may find a large or even record breaking
prime and enter into Chris Caldwell‘s The Largest Known Primes Database as a Titan!
![Page 6: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/6.jpg)
Heart Simulator
● Current computational models track the electro-mechanics of the heart from sub-cellular to the whole-organ level.
● The models allow a better comprehension of important cardiac diseases, such as Ventricular Arrhythmia, Myocarditis, Infarct, Chagas Disease, Diabetes, etc.
● Complicated system of partial and ordinary differential equations -> useful to use the computational power of the grid.
● This application was gridified to the EELA (E-Science Grid Facility for Europe and Latin America) Infrasctructure.
![Page 7: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/7.jpg)
Pierre Auger Experiment● Using GRID for Simulation and Modeling of Ultra High-Energy Cosmic
Rays● Physic Background
○ UHE CR are in energy range 1018-1020 eV○ Detectors are two types (using Cherenkov radiation and ultraviolet
light)● Computation needs
○ Middleware = gLite○ Software packages: Fortran, CORSIKA (COsmic Ray SImulations for
KAscade), Aires (AIRshower Extended Simulations), ROOT, Geant4.. Modeling the particle showers Simulation of particle showers
![Page 8: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/8.jpg)
Genealogical Sorting Index (GSI)The Cummings Laboratory
a statistic to quantify the degree of exclusive ancestry of individuals in labeled groups on a rooted tree.● Open source package● Web interface
1. Upload a tree file.2. View any tree in your input file. Individuals will be in different colors according to their class assignments.3. Assign class labels for the individuals either by uploading a class label assignment file.
![Page 9: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/9.jpg)
● Search for Extra-Terrestrial Inteligence● Analysis of radio signal from space● Distributed coputing among home PCs
○ 150 000 /225 000 active participants / hosts○ BOINC software ○ free CPU/GPU time
![Page 10: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/10.jpg)
Work with gLite
We worked on an educational grid network "edu". This consisted of SEs and CEs which can be retrieved with lcg-info --list-ce(se)
![Page 11: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/11.jpg)
Workload Managment System (WMS)
● The purpose of the WMS - accept user jobs
- assign them to appropriate CE- recover the status of a job- retrieve results of a job
● Each grid job is described using a Job Description Language (JDL) file
● This file specifies the job's characteristics and constraints used to find the most suitable CE
![Page 12: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/12.jpg)
More information about CEs (such as the number of CPUs and the number of jobs being executed) can be obtained with lcg-infosites --vo edu ce
In a similar way we can get more info about SEs - lcg-infosites --vo edu se
![Page 13: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/13.jpg)
Basic job types
● normal job - the most simple type of job
● job collection - submission of a set of jobs whose description files are placed in one directory
● parametric job - submission of a set of jobs having the same descriptions apart from the values of the parametric attributes. Use when your jobs differs only in argument values or input/output files.
● MPI (Message Passing Interface) job
![Page 14: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/14.jpg)
MPI Jobs
● Message Passing Interface - currently the most commonly used interface for the development of parallel applications.
● Parallel applications spawn over several processes which are in the common case mapped over an equivalent number of processors or physical CPUs.
● These processes may communicate with one another by passing information in MPI terminology.
● The goal is to reduce the overall execution time of the application.
![Page 15: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/15.jpg)
Job Submission
● list of CEs that can accept job - glite-wms-job-list-match -a job.jdl
● job.jdl submission - glite-wms-job-submit -a -o job.id job.jdl
● status - glite-wms-job-status -i job.id - Submitted, Waiting, Ready, Scheduled, Running, Done, Cleared, Aborted, Cancelled and Purged
● retrieve results - glite-wms-job-output --dir results -i job.id
![Page 16: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/16.jpg)
Data Management System
● Storing data on different systems induce a need of common interface SRM (Storage Resource Manager)
● On the GRID there are files and replicas of files
● Each file has GUID (Grid Unique IDentifier) and may have one or more LFN aliases (Logical File Name)
● Client always communicates with SE through SRM and keeps track of all his files thanks to LFC (LCG File Catalogue)
![Page 17: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/17.jpg)
Data Management SystemExamples of using:
Get list of files and directories
Then you might want to create new directory
And copy file to SE and register it into catalogue
Or download file from SE
![Page 18: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/18.jpg)
Data Management SystemExamples of using:
You can add new alias for the uploaded file and list all alias
Creating replica to SE for file with GUID
And after work you can detele file
![Page 19: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/19.jpg)
Our experience with GRID
● One really simple task about a data management where we used hostname command
● Managing of a graphic data - resizing of our images (*.jpg, *.png) on grid
● Processing of a video data - converting and resizing of our videos (*.avi, *.flv) with ffmpeg
● Blender's model visualization - rendering of few models
● MPI examples - the value of π, numerical integrationELMER, FDS, MEEP
![Page 20: Introduction to grid technologies and practical usage of ...newuc.jinr.ru/img_sections/file/Practice2013/Presentation_Eu/Stefko... · Or download file from SE. ... Blender's model](https://reader034.vdocuments.pub/reader034/viewer/2022051602/5ae7975c7f8b9a29048f0356/html5/thumbnails/20.jpg)
Thank you for attention!