tat cracow grid workshop, october 27 – 29, 2003 marian bubak, michal turala and the crossgrid...
Post on 21-Dec-2015
214 views
TRANSCRIPT
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Marian Bubak, Michal Turala and the CrossGrid Collaboration
www.eu-crossgrid.org
CrossGrid in Its Halfway: Achievements and Challenges
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Outline
Applications Overview of X# architecture Tool environment Grid middleware X# and OGSA Software development, standards,
bodies, … Summary
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Medical Application
Adding small modifications to the proposed structure results in immediate changes in the blood flow
Online presentation of simulation results via a 3D environment
The progress of the simulation and the estimated time of convergence should be available for inspection
LB flowsimulation
VEWDPC
PDA
Visualization
Interaction
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Basic Features of Flood Simulation Meteorological
• Intensive simulation (HPC), large input/output data sets, high availability of resources
Hydrological• Parametric simulations
(HTC) may require different models (heterogeneous simulations)
Hydraulic• Many 1-D simulations
HTC, 2-D hydraulic simulations require HPC
Data sources
Meteorological simulations
Hydraulic simulations
Hydrological simulations
Users
Output visualization
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Distributed Data Analysis in HEP
Objectives• Access to distributed
databases• Data mining with neural
networks Issues
• Typical interactive requests will run on o(TB) of distributed data
• Transfer/replication may take hours
• Data are transferred before interactive session
• Allocation, installation and setup of the database servers before the interactive session
Replica Manager
Interactive Session Resource Broker
DISTRIBUTEDPROCESSING
DB Installation
Interactive Session
Database server
Interactive Session Manager
Interactive
SessionWorker
InteractiveSession
Worker
InteractiveSessionWorker
Interactive
SessionWorker
InteractiveSession
Worker
PortalXML in/out
On-line output
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Weather Forecast and Air Pollution Modeling
Distributed/parallel code on Grid• Coupled Ocean/Atmosphere Mesoscale
Prediction System• STEM-II Air Pollution Code• Integration of distributed databases
Data mining applied to downscaling weather forecasts
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Interactions in the CrossGrid Applications
Interactive steering Control of a cascade of simulations Distributed data access Parameter study with online results Interactive estimation of simulation time
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Features of Interactive Grid Computing
Run time application control• Performance data on-line
Possible effects of decisions• Access to benchmark information
Interpreting application’s behavior in heterogeneous open system• Access to infrastructure performance
Information meaningful in the context of application field• more application specific performance data
Need in on-line standard and user-defined metrics
TAT
Cracow Grid Workshop, October 27 – 29, 2003
CrossGrid Architecture
Generic Services
Application Specific Services
Supporting Tools
Applications
Collective Layer Libraries
Application
OCM-G
Data Access
Roaming AccessInfrastructure
Monitoring
Scheduling
Agent
Tools
DataGrid Job
Management
DataGrid Data
Management
Benchmarks
Globus
Toolkit
User Interaction
Services
Grid Visualization
Kernel
Portal and
Migrating Desktop
MPICH-G HLA
Generic Services
Application Specific Services
Supporting Tools
Applications
Collective Layer Libraries
Application
OCM-G
Data Access
Roaming AccessInfrastructure
Monitoring
Scheduling
Agent
Tools
DataGrid Job
Management
DataGrid Data
Management
Benchmarks
Globus
Toolkit
User Interaction
Services
Grid Visualization
Kernel
Portal and
Migrating Desktop
MPICH-G HLA
Application
OCM-G
Data Access
Roaming AccessInfrastructure
Monitoring
Scheduling
Agent
Tools
DataGrid Job
Management
DataGrid Data
Management
Benchmarks
Globus
Toolkit
User Interaction
Services
Grid Visualization
Kernel
Portal and
Migrating Desktop
MPICH-G HLA
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Migrating Desktop
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
Simulation Output
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
MARMOT MPI Verification Tool
8 rank 1 performs MPI_Recv
8 Rank 0 is pending!
8 Rank 1 is pending!
WARNING: deadlock detected
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
Performance Prediction Tool
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
G-PM Performance Analysis Tool
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
Jiro-based Infrastructure Monitoring
Site Administrator
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
(Parallel) Application Running
SANTA-G Network Packets Monitoring
Site Administrator
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Migrating Desktop
Multiple Sites
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
Benchmark Running
Benchmark Results
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Handling Multiple Mass Storage Systems
Application
OCM-G
Data Access
Portal and RoamingAccess
InfrastructureMonitoring
Scheduling Agent
DataGrid JobManagement
DataGrid DataManagement
Benchmark
Globus Toolkit
User InteractionServices
Grid VisualizationKernel
Tool
Estimation of Data Access Time for Replica Management
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Sites of the CrossGrid Testbed
UCY NicosiaDEMO Athens
Auth Thessaloniki
CYFRONET Cracow
ICM & IPJ Warsaw
PSNC Poznan
CSIC IFIC Valencia
UAB Barcelona
CSIC-UC IFCA
Santander
CSIC RedIris Madrid
LIP Lisbon
USC Santiago
TCD Dublin
UvA Amsterdam
FZK Karlsruhe
II SAS Bratislava
Géant
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Resources in CG Testbed At each site
• LCFG configuration server• User Interface• Gatekeeper (Computing Element)• Worker Nodes• Storage Element
16 sites:• 115 CPUs (Worker Nodes)• 4 TB (Storage Elements)
Com3
IDC
Worker Nodes GatekeeperStorageElement
UserInterface
ConfigurationMachine(LCFG)
Com3
IDC
Monitoring
ResourceBroker
CA machine(offline)
ReplicaCatalogue
IDC
VirtualOrganization
Data General
MyProxy
Grid services (LIP)• Information Index
• Top MDS Information Server, points to site Information Servers
• Resource Broker• Matchmaking and load balancing scheduler
• Replica Catalogue• Database for physical replica file location
• Certificate Proxy Server• Short lived certificates for long lived processes, used by RB
• Virtual Organization Server• Database for user authentication (CROSSGRID VO)
• Monitoring • Mapcenter: network monitoring system
National Certification Authority machines
TAT
Cracow Grid Workshop, October 27 – 29, 2003
And with Related Grids ...
TAT
Cracow Grid Workshop, October 27 – 29, 2003
X# and OGSA
OGSA – Open Grid Services Architecture• General vision of Grid made of services
OGSI – Open Grid Services Infrastructure• Defined conventions and basic interfaces of
Grid Services• OGSI-compliant Grid Service
Globus Toolkit 3.0• Contains OGSI-implementation• OGSI-based higher level Globus Toolkit
services
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Migration to GT3
Replacing GT2 with GT3 on our testbed More technical than architectural issue CrossGrid uses Globus indirectly (through
EDG) No stable relese of GT3 yet No direct requirements for using GT3
• However experimental installation for Task 1.1
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Migration to OGSA
Functionalities of CrossGrid from the beginning designed as services
Web Service protocols (SOAP) widely used Easy to be made OGSI-compliant
• when required• when stable implementation available
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Tools and OGSA
MARMOT – MPI verification• Independent of OGSA – can use any MPI
implementation
Benchmarks• Can publish benchmark results in Grid
information systems
Performance evaluation and prediction tools• Basically independent of OGSA
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Services Portal and Roaming Access
• RAS uses SOAP protocol to communicate• Little effort to migrate to OGSA
Scheduling Agents• Dependent on EDG and Condor• OGSI-compliant interface can be added
Monitoring• Application monitoring is not going to migrate to OGSA – due
to performance issues, however feasibility studies are performed
• Infrastructure monitoring is compatible with EDG R-GMA that is going to migrate to OGSA
Optimization of Data Access• Ready to migrate, SOAP protocol already used
TAT
Cracow Grid Workshop, October 27 – 29, 2003
• Proceeds through all the development phases
• Requires gathering of quality indicators and producing monthly reports
• Crucial for ensuring quality of code• Currently in progress
Software Development and Testing
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Static Code Metrics
Measured separately for each task:
• amount of new lines of code since last report
• compliance with coding conventions
• count of packages, functions and inner classes
• complexity number (McCabe metric)
• count of formal Javadoc comments per class and method
TAT
Cracow Grid Workshop, October 27 – 29, 2003
X# Management in Short
Main Bodies• Project Coordinator• CG Office• Steering Group• Technical Board• Architecture Team• Quality Assurance• Central Dissemination
Better focus on technicalissues and quality control
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Collaboration with DataGrid
Architecture• interoperability, reuse of MW components• joint proposals to GGF
Interactive applications• methodology, generic structure, grid services• security issues for medical applications
Tools: benchmarks and application monitoring Interoperability of EDG testbeds
• collaboration of testbed support teams• mutual recognition of Certification Authorities• common access/usage policy and procedures• common installation/configuration procedures• joint infrastructure for HEP applications
Software Testing
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Collaboration with Other Grid Projects
GRIDSTART• Active participation in Technical Board meetings• Conferences: Cracow Grid Workshop, Across Grids (Santiago)
– to stimulate exchange of information and personal contacts GLOBUS
• Regular contacts• CrossGrid use cases for OGSA
GridLab• Development of grid applications• Application monitoring• Portals, data access, security • Participation in joint testbeds (SC’2002)
EuroGrid and GRIP• Regular contacts and exchange of information
TAT
Cracow Grid Workshop, October 27 – 29, 2003
Research Areas in CG (about 90 papers)
New paradigms for grid application programming Grid PSE for complex problems Collaborative environments Grid monitoring and benchmarking Monitoring of grid applications Performance of grid applications (analysis and prediction) Grid services for interactive applications Building of Virtual Organizations (tools, rules) Methods and tools for grid middleware deployment In F. Berman, G. Fox, T. Hey (eds): Grid Computing, Wiley 2003 Participation in 6FP projects (EGEE, Oct 15 – 5?) GGF – 2 x BOF, and plans for GGF10 in Frankfurt; ICCS’2004
TAT
Cracow Grid Workshop, October 27 – 29, 2003
www.eu-crossgrid.org
•21 partners•5 Work Packages:•WP1 – CrossGrid Applications•WP2 – Grid Tool Environment •WP3 – New Grid Services•WP4 - International Testbed •WP5 – Project Management