The OptIPuter and Its Applications

Invited Talk Cyberinfrastructure for Humanities, Arts, and Social Sciences

A Summer InstituteSDSC UCSD

July 26, 2006

Dr. Larry Smarr

Director, California Institute for Telecommunications and Information Technologies

Harry E. Gruber Professor,

Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSD

From “Supercomputer–Centric” to “Supernetwork-Centric” Cyberinfrastructure

1.E+00

1.E+01

1.E+02

1.E+03

1.E+04

1.E+05

1.E+06

1985 1990 1995 2000 2005

Ba

nd

wid

th (

Mb

ps

)

Megabit/s

Gigabit/s

Terabit/s

Network Data Source: Timothy Lance, President, NYSERNet

32x10Gb “Lambdas”

1 GFLOP Cray2

60 TFLOP Altix

Bandwidth of NYSERNet Research Network Backbones

T1

Optical WAN Research Bandwidth Has Grown Much Faster Than

Supercomputer Speed!

Co

mp

utin

g S

peed

(G

FL

OP

S)

The OptIPuter Project – Creating a “SuperWeb” for Data Intensive Researchers

• NSF Large Information Technology Research Proposal– Calit2 (UCSD, UCI) and UIC Lead Campuses—Larry Smarr PI– Partners: SDSC, USC, SDSU, NCSA, NW, TA&M, UvA, SARA,

NASA Goddard, KISTI, AIST, CRC(Canada), CICESE (Mexico)

• Industrial Partners– IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent

• $13.5 Million Over Five Years—Now In the Fourth YearNIH Biomedical Informatics

NSF EarthScope and ORIONResearch Network

What is the OptIPuter?

• Applications Drivers Interactive Analysis of Large Data Sets

• OptIPuter Nodes Scalable PC Clusters with Graphics Cards

• IP over Lambda Connectivity Predictable Backplane

• Open Source LambdaGrid Middleware Network is Reservable

• Data Retrieval and Mining Lambda Attached Data Servers

• High Defn. Vis., Collab. SW High Performance Collaboratory

See Nov 2003 Communications of the ACM for Articles on OptIPuter Technologies

www.optiputer.net

fc *

Dedicated Optical Channels Makes High Performance Cyberinfrastructure Possible

(WDM)

Source: Steve Wallach, Chiaro Networks

“Lambdas”Parallel Lambdas are Driving Optical Networking

The Way Parallel Processors Drove 1990s Computing

National Lambda Rail (NLR) and TeraGrid Provides Cyberinfrastructure Backbone for U.S. Researchers

NLR 4 x 10Gb Lambdas Initially Capable of 40 x 10Gb wavelengths at Buildout

Links Two Dozen State and Regional Optical

Networks

DOE, NSF, & NASA

Using NLR

San Francisco Pittsburgh

Cleveland

San Diego

Los Angeles

Portland

Seattle

Pensacola

Baton Rouge

HoustonSan Antonio

Las Cruces /El Paso

Phoenix

New York City

Washington, DC

Raleigh

Jacksonville

Dallas

Tulsa

Atlanta

Kansas City

Denver

Ogden/Salt Lake City

Boise

Albuquerque

UC-TeraGridUIC/NW-Starlight

Chicago

International Collaborators

NSF’s TeraGrid Has 4 x 10Gb Lambda Backbone

Creating a North American Superhighway for High Performance Collaboration

Next Step: Adding Mexico to Canada’s CANARIE and the U.S. National Lambda Rail

OptIPuter Scalable Adaptive Graphics Environment (SAGE) Allows Integration of HD Streams

OptIPortal– Termination

Device for the

OptIPuter Global

Backplane

OptIPortal– Termination Device for the OptIPuter Global Backplane

• 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000• 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC!• Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC

Source: Phil Papadopoulos SDSC, Calit2

The World’s Largest Tiled Display Wall—Calit2@UCI’s HIPerWall

Zeiss Scanning Electron Microscope

Center of Excellence in Calit2@UCI

Albert Yee, PI

Calit2@UCI Apple Tiled Display WallDriven by 25 Dual-Processor G5s

50 Apple 30” Cinema Displays200 Million Pixels of Viewing Real Estate!

HDTV Digital Cameras

Digital Cinema

Falko Kuester and Steve Jenks, PIs

Featured in Apple Computer’s

“Hot News”

3D Videophones Are Here! The Personal Varrier Autostereo Display

• Varrier is a Head-Tracked Autostereo Virtual Reality Display– 30” LCD Widescreen Display with 2560x1600 Native Resolution– A Photographic Film Barrier Screen Affixed to a Glass Panel

– The Barrier Screen Reduces the Horizontal Resolution To 640 Lines

• Cameras Track Face with Neural Net to Locate Eyes• The Display Eliminates the Need to Wear Special Glasses

Source: Daniel Sandin, Thomas DeFanti, Jinghua Ge, Javier Girado, Robert Kooima, Tom Peterka—EVL, UIC

How Do You Get From Your Lab to the National LambdaRail?

www.ctwatch.org

“Research is being stalled by ‘information overload,’ Mr. Bement said, because data from digital instruments are piling up far faster than researchers can study. In particular, he said, campus networks need to be improved. High-speed data lines crossing the nation are the equivalent of six-lane superhighways, he said. But networks at colleges and universities are not so capable. “Those massive conduits are reduced to two-lane roads at most college and university campuses,” he said. Improving cyberinfrastructure, he said, “will transform the capabilities of campus-based scientists.”-- Arden Bement, the director of the National Science Foundation

To Build a Campus Dark Fiber Network—First, Find Out Where All the Campus Conduit Is!

UCSD Campus-Scale Routed OptIPuter with Nodes for Storage, Computation and Visualization

The New Optical Core of the UCSD Campus-Scale Testbed:Evaluating Packet Routing versus Lambda Switching

Goals by 2007:

>= 50 endpoints at 10 GigE

>= 32 Packet switched

>= 32 Switched wavelengths

>= 300 Connected endpoints

Approximately 0.5 TBit/s Arrive at the “Optical” Center

of CampusSwitching will be a Hybrid

Combination of: Packet, Lambda, Circuit --OOO and Packet Switches

Already in Place

Funded by NSF MRI

Grant

Lucent

Glimmerglass

Force10

Created 09-27-2005 by Garrett Hildebrand

Modified 11-03-2005 by Jessica Yu

Calit2 Building

UCInet

10 GE

HIPerWall

LosAngeles

SPDS

Catalyst 3750 in CSI

ONS 15540 WDM at UCI campus MPOE (CPL)

1 GE DWDM Network Line Tustin CENIC Calren

POP

UCSD Optiputer Network

10 GE DWDM Network Line

Engineering Gateway Building,

Catalyst 3750 in 3rd

floor IDF

MDF Catalyst 6500 w/ firewall, 1st floor closet

Wave-2: layer-2 GE. UCSD address space 137.110.247.210-222/28

Floor 2 Catalyst 6500

Floor 3 Catalyst 6500

Floor 4 Catalyst 6500

Wave-1: UCSD address space 137.110.247.242-246 NACS-reserved for testing

ESMFCatalyst 3750 in NACS Machine Room (Optiputer)

Viz Lab

Wave 1 1GEWave 2 1GE

OptIPuter@UCI is Up and Working

Kim-Jitter Measurements

This Week!

Calit2/SDSC Proposal to Create a UC Cyberinfrastructure

of OptIPuter “On-Ramps” to TeraGrid Resources

UC San Francisco

UC San Diego

UC Riverside

UC Irvine

UC Davis

UC Berkeley

UC Santa Cruz

UC Santa Barbara

UC Los Angeles

UC Merced

OptIPuter + CalREN-XD + TeraGrid = “OptiGrid”

Source: Fran Berman, SDSC , Larry Smarr, Calit2

Creating a Critical Mass of End Users on a Secure LambdaGrid

OptIPuter Software Architecture--a Service-Oriented Architecture Integrating Lambdas Into the Grid

GTP XCP UDT

LambdaStreamCEP RBUDP

DVC Configuration

Distributed Virtual Computer (DVC) API

DVC Runtime Library

Globus

XIOGRAM GSI

Distributed Applications/ Web Services

Telescience

Vol-a-Tile

SAGE JuxtaView

Visualization

Data Services

LambdaRAM

DVC Services

DVC Core Services

DVC Job Scheduling

DVCCommunication

Resource Identify/Acquire

NamespaceManagement

Security Management

High SpeedCommunication

Storage Services

IPLambdas

Discovery and Control

PIN/PDC RobuStore

Source: Andrew Chien, UCSD

PI Larry Smarr

Announced January 17, 2006$24.5M Over Seven Years

Marine Genome Sequencing ProjectMeasuring the Genetic Diversity of Ocean Microbes

CAMERA’s Sorcerer II Data Will Double Number of Proteins

in GenBank!

Announced January 17, 2006

Flat FileServerFarm

W E

B P

OR

TA

L

TraditionalUser

Response

Request

DedicatedCompute Farm(1000 CPUs)

TeraGrid: Cyberinfrastructure Backplane(scheduled activities, e.g. all by all comparison)

(10000s of CPUs)

Web(other service)

Local Cluster

LocalEnvironment

DirectAccess LambdaCnxns

Data-BaseFarm

10 GigE Fabric

CAMERA’s Direct Access Core Architecture Will Create Next Generation Metagenomics Server

Source: Phil Papadopoulos, SDSC, Calit2+

We

b S

erv

ice

s

Sargasso Sea Data

Sorcerer II Expedition (GOS)

JGI Community Sequencing Project

Moore Marine Microbial Project

NASA Goddard Satellite Data

Community Microbial Metagenomics Data

The Future Home of the Moore Foundation Funded Marine Microbial Ecology Metagenomics Complex

First Implementation of the CAMERA Complex

Photo Courtesy Joe Keefe, Calit2

Major Buildout of Calit2 Server Room Underway

Calit2 and the Venter Institute Will Combine Telepresence with Remote Interactive Analysis

OptIPuter Visualized

Data

HDTV Over

Lambda

Live Demonstration

of 21st Century National-Scale Team Science 25 Miles

Venter Institute

UIC/UCSD 10GE CAVEWave on the National LambdaRailEmerging OptIPortal Sites

CAVEWave Connects Chicago to Seattle to San Diego…and Washington D.C. as of 4/1/06

and JCVI as of 5/15/06

NEW!

NEW!

SunLight

CICESE

UW

JCVI

MIT

SIO UCSD

SDSU

UIC EVL

UCI

OptIPortals

September 26-30, 2005Calit2 @ University of California, San Diego

California Institute for Telecommunications and Information Technology

Borderless CollaborationBetween Global University Research Centers at 10Gbps

iGrid

2005T H E G L O B A L L A M B D A I N T E G R A T E D F A C I L I T Y

Maxine Brown, Tom DeFanti, Co-Chairs

www.igrid2005.org

100Gb of Bandwidth into the Calit2@UCSD BuildingMore than 150Gb GLIF Transoceanic Bandwidth!450 Attendees, 130 Participating Organizations

20 Countries Driving 49 Demonstrations1- or 10- Gbps Per Demo

CineGrid Leverages OptIPuter Cyberinfrastructure to Enable Global “Extreme Media” Collaboration

• CineGrid Experiments Aim to Push the State of the Art in: – Streaming & Store-and-Forward File Transfer

– Using High-speed, Low-latency Network Protocols

– HDTV in Various Formats for: – Teleconferencing

– Telepresence

– Production

– 2K And 4K Digital Cinema Workflows and Distribution– Stereo

– In High Resolution (2K, 4K)

– Virtual Reality In Higher Resolution (24 Megapixel)

– Distributed Tiled Displays With 20-100 Megapixels– Long term Digital Archiving

• International Workshop:– Tokyo July 2006– Calit2 Dec 2006

Source: Tom DeFanti, Laurin Herr

OptIPuter 4K Telepresence over IP at iGrid 2005 Demonstrated Technical Basis for CineGrid

Keio University President Anzai

UCSD Chancellor Fox

Lays Technical Basis for

Global Digital

Cinema

Sony NTT SGI

New Calit2 Digital Cinema Auditorium

Calit2 Works with CENIC to Provide the California Optical Core for CineGrid

Calit2UCI

USC

SFSU

UCB

Plus, 1Gb and 10Gb Connections to:

• Seattle, Canada, Japan, Asia, Australia, New Zealand

• Chicago, Canada, Japan, Europe, Russia, China

• Tijuana

CineGridTM will Link UCSD/Calit2 and USC School

of Cinema TV with Keio University Research Institute for Digital Media and Content

Extended SoCal OptIPuter to USC

School of Cinema-Television

Calit2UCSD

Prototype of CineGridTM

Digital Archive of Films

Partnering with SFSU’s Institute for

Next Generation Internet

Source: Laurin Herr, Pacific Interface

CineGridTM Project Leader

Calit2 and the Venter Institute Test CineGrid™ with HDTV Movie by John Carter

Live Demonstration of 21st Century Entertainment Delivery June 14, 2006

JCVI

Sony HDTV JH-3

JC Venter Institute Rockville, MD

Calit2 Auditorium

StarLight

Chicago

iGrid 2005Kyoto Nijo Castle

Source: Toppan Printing

Interactive VR Streamed Live from Tokyo to

Calit2 Over Dedicated GigE

and Projected at

4k Resolution

iGrid 2005 Cultural HeritageChina and USA

• Great Wall Cultural Heritage• International Media Centre, China• San Diego State University, USA• Great Wall Society, China• San Diego Supercomputer Center, USA• Chinese Academy of Sciences, China• GLORIAD, USA• Chinese Institute of Surveying and Mapping, China• Cybermapping Lab, University of Texas-Dallas, USA• GEON viz/3D-scanning lab, University of Idaho, USA• Stanford University, USA

www.internationalmediacentre.com/imc/index.html

Data Acquisition from Laser Scanning Combined with Photogrammetry Enables the Construction of Unique Cultural Heritage Images from China.

3D Designs Of The Great Wall Are Combined With 3D Scans Of Physical Images And Satellite Imagery Stored On Servers In China And San Diego.

Source : Maxine Brown, EVL UIC

NSF’s Ocean Observatories Initiative (OOI)Envisions Global, Regional, and Coastal Scales

LEO15 Inset Courtesy of Rutgers University, Institute of Marine and Coastal Sciences

$300M in President’s

Budget for OOI

Coupling Regional and Coastal Ocean Observatories Using OptIPuter and Web/Grid Services

LOOKING: (Laboratory

for the Ocean Observatory Knowledge Integration

Grid)

www.neptune.washington.edu www.mbari.org/mars/

www.sccoos.org/LOOKING Funded by NSF ITR-

John Delaney, UWash, PI)

Using the OptIPuter to Couple Data Assimilation Models to Remote Data Sources Including Biology

Regional Ocean Modeling System (ROMS) http://ourocean.jpl.nasa.gov/

NASA MODIS Mean Primary Productivity for April 2001 in California Current System

Interactive Remote Data and Visualization Services

• Multiple Scalable Displays• Hardware Pixel Streaming• Distributed Collaboration

• Scientific-Info Visualization• AMR Volume Visualization• Glyph and Feature Vis• Visualization Services

•Data Mining for Areas of Interest•Analysis and Feature Extraction•Data Mining Services

NCSA Altix Data and Vis Server

An SDSC/NCSA Data Collaboration

National Laboratory for Advanced Data Research

Linking to OptIPuter

The Synergy of Digital Art and ScienceVisualization of JPL Simulation of Monterey Bay

Source: Donna Cox, Robert Patterson, NCSAFunded by NSF LOOKING Grant

4k Resolution

First Remote Interactive High Definition Video Exploration of Deep Sea Vents-Prototyping NEPTUNE

Source John Delaney & Deborah Kelley, UWash

Canadian-U.S. Collaboration

High Definition Still Frame of Hydrothermal Vent Ecology 2.3 Km Deep

White Filamentous Bacteria on 'Pill Bug' Outer Carapace

1 cm.

Source: John Delaney and

Research Channel, U Washington