RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
1
Ловаш Р (MTA SZTAKI).Афанасьев А.П. (ИСА РАН)Волошинов В.В. ((ИСА РАН)Посыпкин М.А. (ИСА РАН)
Сухорослов О.В. (ИСА РАН)Храпов Н.П. (ИСА РАН)
DEGISCO is supported by the FP7 Capacities Programme under contract nr RI-261561-261561
1
EXPANDING SCIENTIFIC COMPUTATIONAL INFRASTRUCTURES WITH DESKTOP GRIDSEXPANDING SCIENTIFIC COMPUTATIONAL INFRASTRUCTURES WITH DESKTOP GRIDS
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
2
Desktop Grids can be categorized as Desktop Grids can be categorized as a successful type of distributed a successful type of distributed infrastructure that is widely used infrastructure that is widely used in science and industry; both the in science and industry; both the largest Grids for science and in largest Grids for science and in industry are based on this industry are based on this technology.technology.
The presentation provides an The presentation provides an overview of Desktop Grids and their overview of Desktop Grids and their place in the infrastructure world.place in the infrastructure world.
Desktop Grids introductionDesktop Grids introductionDesktop Grids introductionDesktop Grids introduction
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
3RI-261561 WP4version: 1.0
IntroductionAuthor: Robert Lovas, Ad Emmen
http://knowledgebase.e-irg.eu
Why Desktop Grids are Why Desktop Grids are important? important? Why Desktop Grids are Why Desktop Grids are important? important?
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
4
Survey of EDGeS projectSurvey of EDGeS project
Questionnaires all across EuropeQuestionnaires all across Europe
Get an idea of the interest in Get an idea of the interest in people and SMEs to donate people and SMEs to donate computing time for science to a computing time for science to a GridGrid
Get an idea of the interest in Get an idea of the interest in running a Grid inside an SME running a Grid inside an SME
Prelude - what do people at Prelude - what do people at home and SME’s think about home and SME’s think about grid computinggrid computing
Prelude - what do people at Prelude - what do people at home and SME’s think about home and SME’s think about grid computinggrid computing
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
5
Survey amongst the General Survey amongst the General Public and SME’sPublic and SME’sSurvey amongst the General Survey amongst the General Public and SME’sPublic and SME’s
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
Opinions about Grid Opinions about Grid computingcomputingOpinions about Grid Opinions about Grid computingcomputing
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
7
Overall: there is interest in Desktop Grid computing in Europe.
However, that people are willing to change their current practice and say that they want to participate in Grid efforts does not mean that they are actually going to do that.
Need to generate trust in the organization that manages the Grid.
People want to donate computing time for scientific applications, especially medical applications. They do not like to donate computing time to commercial or defense applications.
People want feedback on the application they are running.
No clear technical barriers perceived by the respondents: so this does not need much attention.
Overall the respondents were rather positive about donating computing time for a Grid or about running applications on a Grid.
Survey - ConclusionsSurvey - ConclusionsSurvey - ConclusionsSurvey - Conclusions
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
8
Connecting a large number of resources Connecting a large number of resources and make them available for a set of and make them available for a set of applicationsapplications
Resources belong to different ownersResources belong to different owners
Heterogeneous system (HW, SW, network)Heterogeneous system (HW, SW, network)
Typically for Desktop Grids: it is the Typically for Desktop Grids: it is the Desktop Grid computer that initiates Desktop Grid computer that initiates all communications: there is no central all communications: there is no central control over the desktop computercontrol over the desktop computer
Most computers in a Desktop Grid are Most computers in a Desktop Grid are desktop computers, but they can also be desktop computers, but they can also be notebooks, or servers, or mobile phonesnotebooks, or servers, or mobile phones
A Grid definitionA Grid definitionA Grid definitionA Grid definition
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
9
What is Grid?What is Grid?What is Grid?What is Grid?
Using physical resources belonging Using physical resources belonging to different organisations for a to different organisations for a set of applicationsset of applications
Main problem: crossing Main problem: crossing (administrative) boundaries between (administrative) boundaries between the organisationsthe organisations
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
10
Processors - compute GridProcessors - compute Grid
Storage - storage GridStorage - storage Grid
Data bases, repositories - data Data bases, repositories - data GridGrid
Sensors - sensor GridSensors - sensor Grid
Resources that can be sharedResources that can be sharedResources that can be sharedResources that can be shared
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
11
Citizens’ and SMEs’ PC’s: Desktop Citizens’ and SMEs’ PC’s: Desktop GridGrid
Thousands or even millions of owners
Company divisions: Enterprise GridCompany divisions: Enterprise GridA dozen or so organisations within one enterprize
Universities and research Universities and research institutes - eScience Grid institutes - eScience Grid
Up to a few hundred (in the case of EGEE, a pan-European Grid)
Different owners of Different owners of resourcesresourcesDifferent owners of Different owners of resourcesresources
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
12
Recipe:Recipe:Take resourcesTake resources
Connect them through a fast networkConnect them through a fast network
Add Grid middlewareAdd Grid middleware
And your Grid is readyAnd your Grid is ready
But you have to port your applications to the GridBut you have to port your applications to the Grid
A Grid is an infrastructureA Grid is an infrastructureA Grid is an infrastructureA Grid is an infrastructure
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
13
Almere is a new town in the NetherlandsAlmere is a new town in the Netherlands
Good cradle for the world’s first CityGridGood cradle for the world’s first CityGrid
Desktop Grid used for scientific Desktop Grid used for scientific applications: sharing CPU powerapplications: sharing CPU power
Back-Up Grid experiment to share hard disk Back-Up Grid experiment to share hard disk space (as part of EU BEinGRID project)space (as part of EU BEinGRID project)
Build-on-Grid and Virtual Reality based Build-on-Grid and Virtual Reality based environment to allow people to build their environment to allow people to build their own house in virtual space with all the own house in virtual space with all the right data and checking permits before it right data and checking permits before it is built in reality. (Feasibility study for is built in reality. (Feasibility study for municipality finished.)municipality finished.)
An exampleAn exampleAlmereGrid - World’s first AlmereGrid - World’s first CityGridCityGrid
An exampleAn exampleAlmereGrid - World’s first AlmereGrid - World’s first CityGridCityGrid
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
14
Applications on the GridScience Grid
applications
Build-on-Grid Own house
construction
HealthGrid diagnosis agent based
AlmereGrid infrastructure
e-EducationusingGrid
Strengthen
knowledge infra-
structure of the city
BEinGRIDbackup/ restore
Broadband network
For science and beyondFor science and beyondFor science and beyondFor science and beyond
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
15
AgendaAgendaBasic techniquesBasic techniques
Open source examplesOpen source examplesXtremWebXtremWeb
BOINCBOINC
Commercial examplesCommercial examplesLSF DesktopLSF Desktop
DatasynapseDatasynapse
ParabonParabon
Desktop Grids - Desktop Grids - IntroductionIntroductionDesktop Grids - Desktop Grids - IntroductionIntroduction
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
16
Resource computersResource computersThe resource computers (also called workers, The resource computers (also called workers, clients,..) execute the workclients,..) execute the work
User system or interfaceUser system or interfaceInterface for the Grid user to submit his/her Interface for the Grid user to submit his/her application, monitor the progress, and retrieve application, monitor the progress, and retrieve resultsresults
Grid serverGrid serverHandles the requests from the resource computers to Handles the requests from the resource computers to get new work or submit the resultsget new work or submit the results
Handles the requests from the users to execute jobsHandles the requests from the users to execute jobs
Basic structureBasic structureBasic structureBasic structure
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
17
CompaniesCompanies
Residential Residential areasareas
Universities & research institutesUniversities & research institutes
Grid Grid organisationorganisation
BasicBasic Desktop Desktop Grid Grid (players)(players)BasicBasic Desktop Desktop Grid Grid (players)(players)
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
18
Grid resourcesGrid resources
Grid resourcesGrid resources Grid usersGrid users
Grid serverGrid server
Basic Basic Desktop Desktop Grid Grid (components)(components)Basic Basic Desktop Desktop Grid Grid (components)(components)
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
19
Volunteer Volunteer Desktop Desktop GridsGridsVolunteer Volunteer Desktop Desktop GridsGrids
Uses a pullUses a pull--model: it is always model: it is always the resource computer that asks for the resource computer that asks for work and initiates all work and initiates all communication with the servercommunication with the server
There is an application There is an application repository: users can only run repository: users can only run validated trusted applicationsvalidated trusted applications
The resources, however, are The resources, however, are untrusted and volatile: you are not untrusted and volatile: you are not sure they deliver the correct sure they deliver the correct answeranswer
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
20
Local DeskLocal Deskttop Gridsop GridsLocal DeskLocal Deskttop Gridsop Grids
Organisations Organisations (universities, (universities, companies) can also run an internal companies) can also run an internal Desktop Grid.Desktop Grid.
Can be composed of trusted Can be composed of trusted resourcesresources
Use spare capacity Use spare capacity
Can also include servers and Can also include servers and clustersclusters
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
21
Desktop Grid applicationsDesktop Grid applicationsDesktop Grid applicationsDesktop Grid applications
There is --in general-- no There is --in general-- no communication between the resourcescommunication between the resources
Hence you can only run Hence you can only run a subclass a subclass of parallelof parallel programs: master- programs: master-worker, parameter sweep or monte-worker, parameter sweep or monte-carlocarlo
In volunteer computing the network In volunteer computing the network bandwidth and the HDD/CPU capacity bandwidth and the HDD/CPU capacity can be rather limitedcan be rather limited
But there can be large numbers of But there can be large numbers of resourcesresources
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
22
XtremWebXtremWebXtremWebXtremWeb
Developed by IN2P3 and INRIA in Developed by IN2P3 and INRIA in FranceFrance
http://xtremweb.net
Written in JavaWritten in Java
Can be used for all type of Grids, Can be used for all type of Grids, from simple local to advanced from simple local to advanced volunteer Desktop Gridsvolunteer Desktop Grids
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
23
BOINCBOINCBOINCBOINC
Middleware designed for volunteer Middleware designed for volunteer Desktop grid computingDesktop grid computing
http://boinc.berkeley.edu//
Used for the largest Grid in the Used for the largest Grid in the world: seti@homeworld: seti@home
More than 4 million computers More than 4 million computers world wide have BOINC installedworld wide have BOINC installed
Many other projects use BOINCMany other projects use BOINC
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
24
World Community Grid - IBM World Community Grid - IBM (http://www.worldcommunitygrid.org/) (http://www.worldcommunitygrid.org/) 300.000 PCs300.000 PCs
Leiden Classical Grid - Education on Leiden Classical Grid - Education on GridGrid (http://boinc.gorlaeus.net/) (http://boinc.gorlaeus.net/) 17.000 PCs17.000 PCs
SZTAKI - Hungarian GridSZTAKI - Hungarian Grid(http://www.desktopgrid.hu/) (http://www.desktopgrid.hu/) 70.000 PCs70.000 PCs
AlmereGrid (http://almeregrid.nl) 3.000 AlmereGrid (http://almeregrid.nl) 3.000 PCsPCs
PS3GRID (Based on Playstations) PS3GRID (Based on Playstations) (http://ps3grid.net/)(http://ps3grid.net/)
Some other Desktop GridsSome other Desktop GridsSome other Desktop GridsSome other Desktop Grids
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
25
BOINC StructureBOINC StructureBOINC StructureBOINC Structure
BOINC is BOINC is organised around organised around projects: a project is typically projects: a project is typically one applicationone application
Resource computers (BOINC clients) Resource computers (BOINC clients) can connect to one or more projectscan connect to one or more projects
A Grid server can run more A Grid server can run more projects: but most only have one or projects: but most only have one or perhaps a fewperhaps a few
User interface is integrated with User interface is integrated with the projectthe project
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
26
BOINC installationBOINC installationBOINC installationBOINC installation
Installing a BOINC Grid is not Installing a BOINC Grid is not click and playclick and play
Collection of C++ programs, PHP Collection of C++ programs, PHP scripts and other sources, with scripts and other sources, with many dependencies on operating many dependencies on operating system, the database, and other system, the database, and other toolstools
Easiest way: get one of the Easiest way: get one of the distributions. For instance distributions. For instance the the SZTAKI BOINC distribution.SZTAKI BOINC distribution. This This installs relatively fast if you installs relatively fast if you start with a Debian 4.0 systemstart with a Debian 4.0 system
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
27
BOINC Client on resourceBOINC Client on resourceBOINC Client on resourceBOINC Client on resource
Customisable, easy-to-installCustomisable, easy-to-install
Easy view and complex view with a Easy view and complex view with a lot of details on what is lot of details on what is happeninghappening
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
28
BOINC applicationsBOINC applicationsBOINC applicationsBOINC applications
BOINC applications have to be BOINC applications have to be compiled with calls to a special compiled with calls to a special BOINC API.BOINC API.
There are some wrappers for There are some wrappers for executables, like genwrapper executables, like genwrapper (developed by MTA SZTAKI) (developed by MTA SZTAKI)
More extensive API’s exist for More extensive API’s exist for instance for master worker-instance for master worker-applications: DC-API (developed by applications: DC-API (developed by MTA SZTAKI)MTA SZTAKI)
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
29
Q. A Desktop Grid consists of otherwise Q. A Desktop Grid consists of otherwise unused computing sources. So it is for free?unused computing sources. So it is for free?
A. It is much cheaper than any other solution, but of course if you A. It is much cheaper than any other solution, but of course if you use a computer it uses power and produces heatuse a computer it uses power and produces heat
Q. Are Desktop Grids Green? Q. Are Desktop Grids Green? A. Yes: less computers have to be built. A. Yes: less computers have to be built.
Q. So we do not need supercomputers anymore?Q. So we do not need supercomputers anymore?A. No. Although there are many applications that can run on a A. No. Although there are many applications that can run on a Desktop Grid, some require fast communication between the Desktop Grid, some require fast communication between the processors. For those supercomputers are more appropriateprocessors. For those supercomputers are more appropriate
Q. It took me a long time as a user to get Q. It took me a long time as a user to get used to Glite within EGEE. Do I have to used to Glite within EGEE. Do I have to learn a new Grid technologylearn a new Grid technology
A. No. The EDGeS project provides a seamless bridge between EGEE A. No. The EDGeS project provides a seamless bridge between EGEE and Desktop Gridsand Desktop Grids
Desktop Grid QA Desktop Grid QA Desktop Grid QA Desktop Grid QA
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
International Desktop Grid International Desktop Grid FederationFederationInternational Desktop Grid International Desktop Grid FederationFederation
Main vehicle for dissemination during Main vehicle for dissemination during the project lifetime and for the project lifetime and for continuation after the project continuation after the project
Federation is open for organisations and Federation is open for organisations and individuals in research and industry individuals in research and industry
A member organisationA member organisation
Focal point for DCI/Desktop operators Focal point for DCI/Desktop operators and Application developersand Application developers
Website: discussion fora, events and Website: discussion fora, events and workshopsworkshops
Also supported by EDGI (For European Also supported by EDGI (For European chapter)chapter)
30RI-261561 WP4 2010-04-27
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
31RI-261561 WP4 2010-04-27
The International Desktop Grid The International Desktop Grid Federation and support from Federation and support from DEGISCODEGISCO
The International Desktop Grid The International Desktop Grid Federation and support from Federation and support from DEGISCODEGISCO
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
32RI-261561 WP4 2010-04-27
Project work plan Project work plan Project work plan Project work plan
WP1 ManagementWP1 Management
WP2 Infrastructure supportWP2 Infrastructure supportTechnically support the setting up and operation Technically support the setting up and operation of new desktop grids, especially in ICPC of new desktop grids, especially in ICPC countries. Both local desktop grids, inside countries. Both local desktop grids, inside organisations, and public volunteer desktop grids organisations, and public volunteer desktop grids will be supported.will be supported.
WP3 Application supportWP3 Application supportThe infrastructure that is supported by WP2, will The infrastructure that is supported by WP2, will be used by scientific applications that will be be used by scientific applications that will be run by researchers in ICPC countries or run by researchers in ICPC countries or collaborations including researchers from ICPC collaborations including researchers from ICPC countries and Europe. countries and Europe.
WP4. Communication, dissemination, WP4. Communication, dissemination, and community managementand community management
Create awareness amongst all possible Create awareness amongst all possible stakeholders; recruit new resource providers, new stakeholders; recruit new resource providers, new grid operators, new application developers, and grid operators, new application developers, and new scientific users. It will also disseminate new scientific users. It will also disseminate project results and provide training. project results and provide training.
32
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
33
SummarySummarySummarySummary
Infrastructures like HPC, cluster grids, desktop grids and cloud have their own roles which are complementaryThey should go hand in hand and collaborate instead of competeContact DEGISCO or ISA RAS for more information and collaboration opportunities
Join the International Desktop Grid Federation
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
QUESTIONS ???QUESTIONS ???QUESTIONS ???QUESTIONS ???
34
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
35
Commercial Commercial Desktop Desktop GridsGridsCommercial Commercial Desktop Desktop GridsGrids
Desktop Grids are a big commercial Desktop Grids are a big commercial successsuccess
Many large companies (pharmaceutical, Many large companies (pharmaceutical, engineering, banks,..) have installed big engineering, banks,..) have installed big Desktop GridsDesktop Grids
But they do not tell much about it: it is But they do not tell much about it: it is a competitive advantage and proven a competitive advantage and proven technologytechnology
Several companies provide Desktop Grid Several companies provide Desktop Grid middleware; although some now call in middleware; although some now call in differently because they do not want to differently because they do not want to be associated with the Globus and gLite be associated with the Globus and gLite based Grids that by many are seen as based Grids that by many are seen as failed in the commercial areafailed in the commercial area
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
36
DatasynapseDatasynapseDatasynapseDatasynapse
More than 1 million computers run More than 1 million computers run Datasynape software in the world. Datasynape software in the world.
Making it, probably, the most Making it, probably, the most successful Grid middleware after successful Grid middleware after BOINCBOINC
Acquired by TIBCO (Cloud provider) Acquired by TIBCO (Cloud provider) recently for US$28 millionrecently for US$28 million
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
37
Platform LSF DesktopPlatform LSF DesktopPlatform LSF DesktopPlatform LSF Desktop
Platform computing is involved in Platform computing is involved in job management systems for already job management systems for already many yearsmany years
Platform LSF is mainly focused at Platform LSF is mainly focused at managing systems with many managing systems with many processors and (large) clusters. processors and (large) clusters. This works in push modeThis works in push mode
Addition: LSF Desktop for Windows Addition: LSF Desktop for Windows machines. This works in pull mode machines. This works in pull mode and is used to connect Desktops to and is used to connect Desktops to an LSF clusteran LSF cluster
RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting
38
Frontier Grid platformFrontier Grid platform
Works as a traditional Desktop Works as a traditional Desktop GridGrid
Fancy, easy to use user GUIFancy, easy to use user GUI
ParabonParabonParabonParabon