LASCO08 1

A Multi-Site Virtual Cluster System for Wide Area Networks

Takahiro Hirofuchi, Takeshi Yokoi, Tadashi Ebara, Yusuke Tanimura, Hirotaka Ogawa, Hidetomo Nakada, Yoshio Tanaka, and Satoshi Sekiguchi

National Institute of Advanced Industrial Science and Technology (AIST)

LASCO08 2

What is Virtual Cluster?

Virtualization Virtual Machine, SAN, VLAN Cloud Computing

Get computing powers on demand Virtualized Datacenters, Amazon

EC2 (1VM = $0.1/hour) Virtual cluster

Large-scale computing environment composed of VMs

Allow quick and easy deployment of many virtual nodes

Share physical clusters for multiple applications

Application

Application

Scientific applicationWeb 2.0 servicesEmulation testbed

LASCO08 3

Background

Our virtual cluster system Virtualization toolkit

for physical clusters Easy to install, low cost Allow users to customize virtualization

Problem Physical resource limitation at a single site

Scalability and flexibility => Use multi-site, many-site resources

Exponential increase of management cost How to manage widely-distributed virtual nodes?

LASCO08 4

Research Focus

Multi-site virtual cluster Solve large-scale deployment and configuration

issues of distributed virtual nodes Break resource limitations at a single site Small management cost, easy-to-use UI Quick deployment and easy customization

Contribution Design and implementation Evaluation in WAN environments

The minimum WAN traffic for management Quick configuration time like physical clusters

LASCO08 5

Scenario Comparison(Single-Site Virtualization)

Application Application

Software deployment and configuration are very tiresome work!• Manage many virtual nodes distributed in many sites• Update and customize them day by day

Site A Site B

LASCO08 6

Scenario Comparison(Multi-Site Cluster Virtualization)

Application

Software deployment and configuration are performed through the view of a single cluster• Many virtual nodes at many sites can be seen like one physical cluster.• Easy management like a local single cluster

Site A Site B

Application

Scientific applicationWeb 2.0 servicesEmulation testbed

Deploymentand configuration

LASCO08 7

Requirement

Powerful management support for large-scale virtual nodes Scalable and robust Flexible customization of node by node Support heterogeneous resources in many sites

Intuitive administration interface Allow quick deployment, minimize

administrative network traffic

LASCO08 8

Design Criteria

Exploit an existing cluster management system for distributed virtual nodes Designed for physical clusters Virtual node management (add, delete, and configure)

Integrate distributed VMs by Ethernet VPN Bridge internal networks of single-site virtual clusters Transparency for admins and users

Transparent package caching at each site Package-based software installer Quick installation and reconfiguration Flexible customizability

LASCO08 9

Cluster Management System(NPACI Rocks)

FrontendNode

ComputeNode

Private Network （ LAN）

InstallationConfigurationOperation

Node Config.

Database

• Automatic node install by PXE network boot• Cluster-wide application package called “Roll”• Parallel command execution tools (e.g., cluster-fork, tentakel)• Node status monitoring (e.g., Ganglia)• Automatic recovery from node down

LASCO08 10

Virtual Cluster Management

VirtualFrontend

VirtualNode

Allocate a group of VMsIsolate them by VLANEthernet

VPN

Private Network

InstallationConfigurationOperation

Node Config.

Database

LASCO08 11

Virtual Node (Re-)Configuration(1)

VirtualFrontend

VirtualNode

EthernetVPNNode Config. DB

NodeA ：　 App P ＆ Conf X

NodeB ：　 App Q ＆ Conf Y

NodeC ：　 App R ＆ Conf Z…

PXE Boot Installation

Package ＆ ConfigurationDownload

LASCO08 12

Virtual Node (Re-)Configuration(2)

VirtualFrontend

VirtualNode

EthernetVPNNode Config. DB

NodeA ：　 App P ＆ Conf X

NodeB ：　 App Q ＆ Conf Y

NodeC ：　 App R ＆ Conf Z…

PXE Boot Installation

Package ＆ ConfigurationDownload

Huge Download TrafficPackages x #Nodes

LASCO08 13

Virtual Node (Re-)Configurationwith Package Caching

VirtualFrontend

VirtualNode

EthernetVPN

PackageCacheServer

Minimize admin. trafficfor node configuration

Node Config. DBNodeA ：　 App P ＆ Co

nf XNodeB ：　 App Q ＆ Co

nf YNodeC ：　 App R ＆ Co

nf Z…

LASCO08 14

PrototypeImplementation

• REST API Create a virtual cluster　 Add/Delete a VM Start/Stop a VPN• Reservation portal Site resource monitoring Automatic reservation

• NPACI Rocks 4.2• VMware Server• VLAN, iSCSI• OpenVPN 2.0• Squid 3.0

LASCO08 15

Evaluation

Current status Distributed VM configuration Parallel command line tools Node status monitoring

Points of evaluation Feasibility

WAN environment Large number of virtual nodes

Reconfiguration time Network traffic

Work fine !

LASCO08 16

Experiment Setting

AMD Opteron 244, 3GB Mem, Gb Eth x2

16 nodes 10 or 134 nodes, reconfigured with 900MByte packages

AMD Opteron 246, 6GB Mem, Gb Eth x2

Node Config. DB

LASCO08 17

Experiment without Package Caching

RTT (ms)

Installation Time over WAN (10 Nodes)

(s)

Even for only 10 nodes,it is too slow due to WAN traffic.

LASCO08 18

Experiment with Package CachingVPN Throughput

Cache ServerThroughput

• RTT 20ms

8MB/s

80MB/s

The package cache server reduces download traffic to a unique set of installed packages (800MB).

22 min

LASCO08 19

Summary

Minimized WAN traffic A unique set of installed packages Pre-caching is possible

Rapid configuration of virtual nodes 20 – 30 min / 134 nodes (RTT20ms) Not so far from local physical nodes

15 – 25 min / 134 nodes

LASCO08 20

Related Work

Virtual cluster + VPN Virtuoso [Northwestern Univ., 2003]

Remote VM access via VPN

VioCluster [Purdue Univ., 2005] Automatic shrink/expand by VPN topologies

Virtual Workspace Globus-based virtual cluster services

Amazon EC2 Xen-image-based deployment

PlanetLab Distributed network emulation testbed

LASCO08 21

Conclusion

Large-scale application environment Scalability beyond a single site Smart management framework Flexible configuration

Multi-site virtual cluster Exploit a powerful cluster manager Single cluster view by Ethernet VPN Package caching at each site

Evaluation Quick configuration of distributed VMs Minimized WAN traffic

LASCO08 22

Future Work

Ongoing Project Xen support VM migration over WAN Diskless boot Other OS support

Availability http://code.google.com/p/grivon

LASCO08 23

Node Installation Progress

Node ID

Time

Generating nodeconfiguration scripts.

Download anInstaller image

Downloadpackages

22 min