Integration of Cloud Computing and Cloud Storage

Overview and Introduction

Patrick Dreher

Chief Scientist ,Renaissance Computing Institute

Adjunct Prof. Computer Science, NC State University

IEEE Mass Storage Conference Tutorial

May 3, 2010

Level of General Interest in Cloud Computing

Where Would One Look To Find Cloud Computing?

CloudTest/Soasta

Plus Numerous Others…….

So many choices…

Picking a Cloud Option is Only The First Step

• Data are not always located near the computation and analysis infrastructure– There is data available ….large quantities of data …

– In many geographically different places…

– With millions of files holding that data

• Goal is to extract useful information from data

• These large data collections need computational systems for analysis

• One option is to move the data to the compute location

Aug 1990100

GBy/mo

Oct 1993

1 TBy/mo Jul

199810

TBy/mo

Nov 2001100

TBy/mo

Apr 2006

1 PBy/m

o

ESnet Traffic Increases by10X Every 47 Months, on Average

Observation of Current and Historical ESnet Traffic PatternsTe

rab

ytes

/ m

on

th

Log Plot of ESnet Monthly Accepted Traffic, January 1990 – June 2009

Courtesy of William Johnston, LBL

Actual volume for Jun 2009: 4.3 Petabytes/month

Projected volume for Jun 2010: 8.6 Petabytes/month

y = 2.3747e0.5714x

y = 0.4511e0.5244x

y = 0.1349e0.4119x

y = 0.8699e0.6704x

0

1

10

100

1000

10000

100000

1000000

10000000

100000000Ja

n, 9

0

Jan

, 91

Jan

, 92

Jan

, 93

Jan

, 94

Jan

, 95

Jan

, 96

Jan

, 97

Jan

, 98

Jan

, 99

Jan

, 00

Jan

, 01

Jan

, 02

Jan

, 03

Jan

, 04

Jan

, 05

Jan

, 06

Jan

, 07

Jan

, 08

Jan

, 09

Jan

, 10

Jan

, 11

Jan

, 12

Jan

, 13

Jan

, 14

Jan

, 15

ESnet traffic

HEP exp. data

ESnet capacity

Climate modeling data

Expon. (ESnet traffic)

Expon. (HEP exp. data)

Expon. (ESnet capacity)

Expon. (Climate modeling data)

2010 value

xx

40 Pby

xx

4 Pby

Network Traffic, Science Data, and Network Capacity

Long-term trends

HistoricalProjection

All

Fou

r D

ata

Seri

es

are

No

rmal

ize

d t

o “

1”

at J

an. 1

99

0 2010 value

--

40 PBy

--

4 PBy

(HEP data courtesy of Harvey Newman, Caltech, and Richard Mount, SLAC. Climate data courtesy Dean Williams, LLNL, and the Earth Systems Grid Development Team.)

Courtesy of William Johnston, LBL

In The Tutorial This Afternoon

• What are the types of options for cloud computing

• How does one select a cloud computing option from among the numerous choices

• What are the important design questions to ask when constructing a coherent cyberinfrastructure of computing and data

Data GridsPolicy-based Data Management

SensorsCloud

Compute

External Repositories

Cloud

Storage Cache

Message Bus

Aggregate sensor

data in cache

SuperComputer

Event DetectionRemote

locations

Simulations

Digital LibraryArchive

Clients

Remote

Users

Data GridMultiple Protocols

Questions