ri-261561 wp4 2010-09-08 asia 1-epikh school for application porting 1 Ловаш Р (mta sztaki)....

RI-261561 WP4 2010-09-08 Asia 1-EPIKH School for Application Porting

1

Ловаш Р (MTA SZTAKI).Афанасьев А.П. (ИСА РАН)Волошинов В.В. ((ИСА РАН)Посыпкин М.А. (ИСА РАН)

Сухорослов О.В. (ИСА РАН)Храпов Н.П. (ИСА РАН)

DEGISCO is supported by the FP7 Capacities Programme under contract nr RI-261561-261561

1

EXPANDING SCIENTIFIC COMPUTATIONAL INFRASTRUCTURES WITH DESKTOP GRIDSEXPANDING SCIENTIFIC COMPUTATIONAL INFRASTRUCTURES WITH DESKTOP GRIDS


2

Desktop Grids can be categorized as Desktop Grids can be categorized as a successful type of distributed a successful type of distributed infrastructure that is widely used infrastructure that is widely used in science and industry; both the in science and industry; both the largest Grids for science and in largest Grids for science and in industry are based on this industry are based on this technology.technology.

The presentation provides an The presentation provides an overview of Desktop Grids and their overview of Desktop Grids and their place in the infrastructure world.place in the infrastructure world.

Desktop Grids introductionDesktop Grids introductionDesktop Grids introductionDesktop Grids introduction


3RI-261561 WP4version: 1.0

IntroductionAuthor: Robert Lovas, Ad Emmen

http://knowledgebase.e-irg.eu

Why Desktop Grids are Why Desktop Grids are important? important? Why Desktop Grids are Why Desktop Grids are important? important?


4

Survey of EDGeS projectSurvey of EDGeS project

Questionnaires all across EuropeQuestionnaires all across Europe

Get an idea of the interest in Get an idea of the interest in people and SMEs to donate people and SMEs to donate computing time for science to a computing time for science to a GridGrid

Get an idea of the interest in Get an idea of the interest in running a Grid inside an SME running a Grid inside an SME

Prelude - what do people at Prelude - what do people at home and SME’s think about home and SME’s think about grid computinggrid computing

Prelude - what do people at Prelude - what do people at home and SME’s think about home and SME’s think about grid computinggrid computing


5

Survey amongst the General Survey amongst the General Public and SME’sPublic and SME’sSurvey amongst the General Survey amongst the General Public and SME’sPublic and SME’s


Opinions about Grid Opinions about Grid computingcomputingOpinions about Grid Opinions about Grid computingcomputing


7

Overall: there is interest in Desktop Grid computing in Europe.

However, that people are willing to change their current practice and say that they want to participate in Grid efforts does not mean that they are actually going to do that.

Need to generate trust in the organization that manages the Grid.

People want to donate computing time for scientific applications, especially medical applications. They do not like to donate computing time to commercial or defense applications.

People want feedback on the application they are running.

No clear technical barriers perceived by the respondents: so this does not need much attention.

Overall the respondents were rather positive about donating computing time for a Grid or about running applications on a Grid.

Survey - ConclusionsSurvey - ConclusionsSurvey - ConclusionsSurvey - Conclusions


8

Connecting a large number of resources Connecting a large number of resources and make them available for a set of and make them available for a set of applicationsapplications

Resources belong to different ownersResources belong to different owners

Heterogeneous system (HW, SW, network)Heterogeneous system (HW, SW, network)

Typically for Desktop Grids: it is the Typically for Desktop Grids: it is the Desktop Grid computer that initiates Desktop Grid computer that initiates all communications: there is no central all communications: there is no central control over the desktop computercontrol over the desktop computer

Most computers in a Desktop Grid are Most computers in a Desktop Grid are desktop computers, but they can also be desktop computers, but they can also be notebooks, or servers, or mobile phonesnotebooks, or servers, or mobile phones

A Grid definitionA Grid definitionA Grid definitionA Grid definition


9

What is Grid?What is Grid?What is Grid?What is Grid?

Using physical resources belonging Using physical resources belonging to different organisations for a to different organisations for a set of applicationsset of applications

Main problem: crossing Main problem: crossing (administrative) boundaries between (administrative) boundaries between the organisationsthe organisations


10

Processors - compute GridProcessors - compute Grid

Storage - storage GridStorage - storage Grid

Data bases, repositories - data Data bases, repositories - data GridGrid

Sensors - sensor GridSensors - sensor Grid

Resources that can be sharedResources that can be sharedResources that can be sharedResources that can be shared


11

Citizens’ and SMEs’ PC’s: Desktop Citizens’ and SMEs’ PC’s: Desktop GridGrid

Thousands or even millions of owners

Company divisions: Enterprise GridCompany divisions: Enterprise GridA dozen or so organisations within one enterprize

Universities and research Universities and research institutes - eScience Grid institutes - eScience Grid

Up to a few hundred (in the case of EGEE, a pan-European Grid)

Different owners of Different owners of resourcesresourcesDifferent owners of Different owners of resourcesresources


12

Recipe:Recipe:Take resourcesTake resources

Connect them through a fast networkConnect them through a fast network

Add Grid middlewareAdd Grid middleware

And your Grid is readyAnd your Grid is ready

But you have to port your applications to the GridBut you have to port your applications to the Grid

A Grid is an infrastructureA Grid is an infrastructureA Grid is an infrastructureA Grid is an infrastructure


13

Almere is a new town in the NetherlandsAlmere is a new town in the Netherlands

Good cradle for the world’s first CityGridGood cradle for the world’s first CityGrid

Desktop Grid used for scientific Desktop Grid used for scientific applications: sharing CPU powerapplications: sharing CPU power

Back-Up Grid experiment to share hard disk Back-Up Grid experiment to share hard disk space (as part of EU BEinGRID project)space (as part of EU BEinGRID project)

Build-on-Grid and Virtual Reality based Build-on-Grid and Virtual Reality based environment to allow people to build their environment to allow people to build their own house in virtual space with all the own house in virtual space with all the right data and checking permits before it right data and checking permits before it is built in reality. (Feasibility study for is built in reality. (Feasibility study for municipality finished.)municipality finished.)

An exampleAn exampleAlmereGrid - World’s first AlmereGrid - World’s first CityGridCityGrid

An exampleAn exampleAlmereGrid - World’s first AlmereGrid - World’s first CityGridCityGrid


14

Applications on the GridScience Grid

applications

Build-on-Grid Own house

construction

HealthGrid diagnosis agent based

AlmereGrid infrastructure

e-EducationusingGrid

Strengthen

knowledge infra-

structure of the city

BEinGRIDbackup/ restore

Broadband network

For science and beyondFor science and beyondFor science and beyondFor science and beyond


15

AgendaAgendaBasic techniquesBasic techniques

Open source examplesOpen source examplesXtremWebXtremWeb

BOINCBOINC

Commercial examplesCommercial examplesLSF DesktopLSF Desktop

DatasynapseDatasynapse

ParabonParabon

Desktop Grids - Desktop Grids - IntroductionIntroductionDesktop Grids - Desktop Grids - IntroductionIntroduction


16

Resource computersResource computersThe resource computers (also called workers, The resource computers (also called workers, clients,..) execute the workclients,..) execute the work

User system or interfaceUser system or interfaceInterface for the Grid user to submit his/her Interface for the Grid user to submit his/her application, monitor the progress, and retrieve application, monitor the progress, and retrieve resultsresults

Grid serverGrid serverHandles the requests from the resource computers to Handles the requests from the resource computers to get new work or submit the resultsget new work or submit the results

Handles the requests from the users to execute jobsHandles the requests from the users to execute jobs

Basic structureBasic structureBasic structureBasic structure


17

CompaniesCompanies

Residential Residential areasareas

Universities & research institutesUniversities & research institutes

Grid Grid organisationorganisation

BasicBasic Desktop Desktop Grid Grid (players)(players)BasicBasic Desktop Desktop Grid Grid (players)(players)


18

Grid resourcesGrid resources

Grid resourcesGrid resources Grid usersGrid users

Grid serverGrid server

Basic Basic Desktop Desktop Grid Grid (components)(components)Basic Basic Desktop Desktop Grid Grid (components)(components)


19

Volunteer Volunteer Desktop Desktop GridsGridsVolunteer Volunteer Desktop Desktop GridsGrids

Uses a pullUses a pull--model: it is always model: it is always the resource computer that asks for the resource computer that asks for work and initiates all work and initiates all communication with the servercommunication with the server

There is an application There is an application repository: users can only run repository: users can only run validated trusted applicationsvalidated trusted applications

The resources, however, are The resources, however, are untrusted and volatile: you are not untrusted and volatile: you are not sure they deliver the correct sure they deliver the correct answeranswer


20

Local DeskLocal Deskttop Gridsop GridsLocal DeskLocal Deskttop Gridsop Grids

Organisations Organisations (universities, (universities, companies) can also run an internal companies) can also run an internal Desktop Grid.Desktop Grid.

Can be composed of trusted Can be composed of trusted resourcesresources

Use spare capacity Use spare capacity

Can also include servers and Can also include servers and clustersclusters


21

Desktop Grid applicationsDesktop Grid applicationsDesktop Grid applicationsDesktop Grid applications

There is --in general-- no There is --in general-- no communication between the resourcescommunication between the resources

Hence you can only run Hence you can only run a subclass a subclass of parallelof parallel programs: master- programs: master-worker, parameter sweep or monte-worker, parameter sweep or monte-carlocarlo

In volunteer computing the network In volunteer computing the network bandwidth and the HDD/CPU capacity bandwidth and the HDD/CPU capacity can be rather limitedcan be rather limited

But there can be large numbers of But there can be large numbers of resourcesresources


22

XtremWebXtremWebXtremWebXtremWeb

Developed by IN2P3 and INRIA in Developed by IN2P3 and INRIA in FranceFrance

http://xtremweb.net

Written in JavaWritten in Java

Can be used for all type of Grids, Can be used for all type of Grids, from simple local to advanced from simple local to advanced volunteer Desktop Gridsvolunteer Desktop Grids


23

BOINCBOINCBOINCBOINC

Middleware designed for volunteer Middleware designed for volunteer Desktop grid computingDesktop grid computing

http://boinc.berkeley.edu//

Used for the largest Grid in the Used for the largest Grid in the world: seti@homeworld: seti@home

More than 4 million computers More than 4 million computers world wide have BOINC installedworld wide have BOINC installed

Many other projects use BOINCMany other projects use BOINC


24

World Community Grid - IBM World Community Grid - IBM (http://www.worldcommunitygrid.org/) (http://www.worldcommunitygrid.org/) 300.000 PCs300.000 PCs

Leiden Classical Grid - Education on Leiden Classical Grid - Education on GridGrid (http://boinc.gorlaeus.net/) (http://boinc.gorlaeus.net/) 17.000 PCs17.000 PCs

SZTAKI - Hungarian GridSZTAKI - Hungarian Grid(http://www.desktopgrid.hu/) (http://www.desktopgrid.hu/) 70.000 PCs70.000 PCs

AlmereGrid (http://almeregrid.nl) 3.000 AlmereGrid (http://almeregrid.nl) 3.000 PCsPCs

PS3GRID (Based on Playstations) PS3GRID (Based on Playstations) (http://ps3grid.net/)(http://ps3grid.net/)

Some other Desktop GridsSome other Desktop GridsSome other Desktop GridsSome other Desktop Grids


25

BOINC StructureBOINC StructureBOINC StructureBOINC Structure

BOINC is BOINC is organised around organised around projects: a project is typically projects: a project is typically one applicationone application

Resource computers (BOINC clients) Resource computers (BOINC clients) can connect to one or more projectscan connect to one or more projects

A Grid server can run more A Grid server can run more projects: but most only have one or projects: but most only have one or perhaps a fewperhaps a few

User interface is integrated with User interface is integrated with the projectthe project


26

BOINC installationBOINC installationBOINC installationBOINC installation

Installing a BOINC Grid is not Installing a BOINC Grid is not click and playclick and play

Collection of C++ programs, PHP Collection of C++ programs, PHP scripts and other sources, with scripts and other sources, with many dependencies on operating many dependencies on operating system, the database, and other system, the database, and other toolstools

Easiest way: get one of the Easiest way: get one of the distributions. For instance distributions. For instance the the SZTAKI BOINC distribution.SZTAKI BOINC distribution. This This installs relatively fast if you installs relatively fast if you start with a Debian 4.0 systemstart with a Debian 4.0 system


27

BOINC Client on resourceBOINC Client on resourceBOINC Client on resourceBOINC Client on resource

Customisable, easy-to-installCustomisable, easy-to-install

Easy view and complex view with a Easy view and complex view with a lot of details on what is lot of details on what is happeninghappening


28

BOINC applicationsBOINC applicationsBOINC applicationsBOINC applications

BOINC applications have to be BOINC applications have to be compiled with calls to a special compiled with calls to a special BOINC API.BOINC API.

There are some wrappers for There are some wrappers for executables, like genwrapper executables, like genwrapper (developed by MTA SZTAKI) (developed by MTA SZTAKI)

More extensive API’s exist for More extensive API’s exist for instance for master worker-instance for master worker-applications: DC-API (developed by applications: DC-API (developed by MTA SZTAKI)MTA SZTAKI)


29

Q. A Desktop Grid consists of otherwise Q. A Desktop Grid consists of otherwise unused computing sources. So it is for free?unused computing sources. So it is for free?

A. It is much cheaper than any other solution, but of course if you A. It is much cheaper than any other solution, but of course if you use a computer it uses power and produces heatuse a computer it uses power and produces heat

Q. Are Desktop Grids Green? Q. Are Desktop Grids Green? A. Yes: less computers have to be built. A. Yes: less computers have to be built.

Q. So we do not need supercomputers anymore?Q. So we do not need supercomputers anymore?A. No. Although there are many applications that can run on a A. No. Although there are many applications that can run on a Desktop Grid, some require fast communication between the Desktop Grid, some require fast communication between the processors. For those supercomputers are more appropriateprocessors. For those supercomputers are more appropriate

Q. It took me a long time as a user to get Q. It took me a long time as a user to get used to Glite within EGEE. Do I have to used to Glite within EGEE. Do I have to learn a new Grid technologylearn a new Grid technology

A. No. The EDGeS project provides a seamless bridge between EGEE A. No. The EDGeS project provides a seamless bridge between EGEE and Desktop Gridsand Desktop Grids

Desktop Grid QA Desktop Grid QA Desktop Grid QA Desktop Grid QA


International Desktop Grid International Desktop Grid FederationFederationInternational Desktop Grid International Desktop Grid FederationFederation

Main vehicle for dissemination during Main vehicle for dissemination during the project lifetime and for the project lifetime and for continuation after the project continuation after the project

Federation is open for organisations and Federation is open for organisations and individuals in research and industry individuals in research and industry

A member organisationA member organisation

Focal point for DCI/Desktop operators Focal point for DCI/Desktop operators and Application developersand Application developers

Website: discussion fora, events and Website: discussion fora, events and workshopsworkshops

Also supported by EDGI (For European Also supported by EDGI (For European chapter)chapter)

30RI-261561 WP4 2010-04-27


31RI-261561 WP4 2010-04-27

The International Desktop Grid The International Desktop Grid Federation and support from Federation and support from DEGISCODEGISCO

The International Desktop Grid The International Desktop Grid Federation and support from Federation and support from DEGISCODEGISCO


32RI-261561 WP4 2010-04-27

Project work plan Project work plan Project work plan Project work plan

WP1 ManagementWP1 Management

WP2 Infrastructure supportWP2 Infrastructure supportTechnically support the setting up and operation Technically support the setting up and operation of new desktop grids, especially in ICPC of new desktop grids, especially in ICPC countries. Both local desktop grids, inside countries. Both local desktop grids, inside organisations, and public volunteer desktop grids organisations, and public volunteer desktop grids will be supported.will be supported.

WP3 Application supportWP3 Application supportThe infrastructure that is supported by WP2, will The infrastructure that is supported by WP2, will be used by scientific applications that will be be used by scientific applications that will be run by researchers in ICPC countries or run by researchers in ICPC countries or collaborations including researchers from ICPC collaborations including researchers from ICPC countries and Europe. countries and Europe.

WP4. Communication, dissemination, WP4. Communication, dissemination, and community managementand community management

Create awareness amongst all possible Create awareness amongst all possible stakeholders; recruit new resource providers, new stakeholders; recruit new resource providers, new grid operators, new application developers, and grid operators, new application developers, and new scientific users. It will also disseminate new scientific users. It will also disseminate project results and provide training. project results and provide training.

32


33

SummarySummarySummarySummary

Infrastructures like HPC, cluster grids, desktop grids and cloud have their own roles which are complementaryThey should go hand in hand and collaborate instead of competeContact DEGISCO or ISA RAS for more information and collaboration opportunities

Join the International Desktop Grid Federation


QUESTIONS ???QUESTIONS ???QUESTIONS ???QUESTIONS ???

34


35

Commercial Commercial Desktop Desktop GridsGridsCommercial Commercial Desktop Desktop GridsGrids

Desktop Grids are a big commercial Desktop Grids are a big commercial successsuccess

Many large companies (pharmaceutical, Many large companies (pharmaceutical, engineering, banks,..) have installed big engineering, banks,..) have installed big Desktop GridsDesktop Grids

But they do not tell much about it: it is But they do not tell much about it: it is a competitive advantage and proven a competitive advantage and proven technologytechnology

Several companies provide Desktop Grid Several companies provide Desktop Grid middleware; although some now call in middleware; although some now call in differently because they do not want to differently because they do not want to be associated with the Globus and gLite be associated with the Globus and gLite based Grids that by many are seen as based Grids that by many are seen as failed in the commercial areafailed in the commercial area


36

DatasynapseDatasynapseDatasynapseDatasynapse

More than 1 million computers run More than 1 million computers run Datasynape software in the world. Datasynape software in the world.

Making it, probably, the most Making it, probably, the most successful Grid middleware after successful Grid middleware after BOINCBOINC

Acquired by TIBCO (Cloud provider) Acquired by TIBCO (Cloud provider) recently for US$28 millionrecently for US$28 million


37

Platform LSF DesktopPlatform LSF DesktopPlatform LSF DesktopPlatform LSF Desktop

Platform computing is involved in Platform computing is involved in job management systems for already job management systems for already many yearsmany years

Platform LSF is mainly focused at Platform LSF is mainly focused at managing systems with many managing systems with many processors and (large) clusters. processors and (large) clusters. This works in push modeThis works in push mode

Addition: LSF Desktop for Windows Addition: LSF Desktop for Windows machines. This works in pull mode machines. This works in pull mode and is used to connect Desktops to and is used to connect Desktops to an LSF clusteran LSF cluster


38

Frontier Grid platformFrontier Grid platform

Works as a traditional Desktop Works as a traditional Desktop GridGrid

Fancy, easy to use user GUIFancy, easy to use user GUI

ParabonParabonParabonParabon


39

And there are more...And there are more...And there are more...And there are more...

Fura: Grid systems, SpainFura: Grid systems, Spain

ProActive: ActiveEon, FranceProActive: ActiveEon, France

Digipede, USA,Digipede, USA,

……

ri-261561 wp4 2010-09-08 asia 1-epikh school for application porting 1 Ловаш Р (mta sztaki)....

Documents