big data and cloud
DESCRIPTION
SomeTRANSCRIPT
Big Data and Cloud
Jun 30, 2011Schubert Zhang
Who am I
• Schubert Zhang ( 张松波 )
• Chief Architect and Director of Big Data Engineering and Cloud
• Research Cloud Tech., Develop Cloud Projects and Products from 2007
• Led the core development team of CMCC “Big Cloud”. @Hanborq
• 10-years telecom products development and tech-management. @UTStarcom
Agenda
• Introduction of Cloud Storage and Computing
• Big Data and Cloud
• Our Big-Data/Cloud Products and Solutions
• Anything for Discussion …
INTRODUCTION OF CLOUD STORAGE AND COMPUTING
PART-1:
A Popular Definition of Cloud …• Cloud computing is a model for enabling convenient, on-demand network access
to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
• Cloud storage is a model of networked online storage where data is stored on multiple servers. Hosting companies operate large data centers, which provides the resources according to the requirements of the customer and expose them as storage pools, which the customers can themselves use to store files or data objects. Physically, the resource may span across multiple servers or/and data centers.
• It promotes availability and is composed of five essential characteristics, three service models, and four deployment models.
A Popular Definition of Cloud …
CommunityCloud
Private Cloud
Public Cloud
Hybrid Clouds
DeploymentModels
ServiceModels
EssentialCharacteristics
Common Characteristics
Software as a Service (SaaS)
Platform as a Service (PaaS)
Infrastructure as a Service (IaaS)
Massive Scale Elastic Computing
Homogeneity Geographic Distribution
Virtualization Service Orientation
Low Cost Software Advanced Security
On Demand Self-Service
Broad Network Access Rapid Elasticity
Resource Pooling Measured Service
Examples of Famous Cloud Products• Google
– Google AppEngine (Storage for Database, etc.)– Google Storage (Storage for Objects)
• Amazon AWS– Simple Storage Service – S3 (Storage for Objects)– Cloud Drive (Online Storage for Individuals)– SimpleDB (Storage for Database)– Elastic Compute Cloud – EC2 (Compute)
• Rackspace– Cloud Servers (Compute)– Cloud Files (Storage for Objects)
• Facebook– Messages– Photo Storage
• Cloudera– Hadoop …
Techs: GFS2/Bigtable/MapReduce/ Megastore/Spanner/Pregel /Dremel…
Techs: Web-Service-Protocol/ Bitstore/Keymap/Dynamo …
Techs: Open Stack …
Techs: Hive/Scribe/Haystack/Hadoop…
We focus on The Technologies Back of the Cloud
• Storage• High Scalability
– Shared-Nothing– Object-Oriented– NoSQL– …
• High Availability– Failure-Detecting– Server Clustering– Replication– Eventual Consistency– …
• Big Data– PB level storage– Structured or non-structured– Information Retrieval– Indexing– Automatic re-sharding/re-partitioning– Automatic load balancing– …
• High Throughput/Latency– Optimized IO and data write/read models.
• Computing• High Scalability• Parallel Computing Framework
– MR - MapReduce– BSP - Bulk Synchronous Parallel
• Job/Task scheduler• Failure rework• PDM - Parallel Data Analysis/Mining
Algorithms– Simple Statistic/Analysis– Classification/Clustering …– For Recommendation and AD– …
BIG DATA AND CLOUDPART-2:
Big Data
• Immutable Law of Big Data– Volume– Variety– Velocity
• Need ….– Distributed System
• Many-many commodity machines
– Scale-out vs. Scale-Up• Scale-out: Auto vs. Manually
Big Data, Big Business$2.25B
$2.35B
$400M
$1.7B
$263M
$250M
>>$30.5M (vc)
Storage Products/SolutionsNAS (Limited Scale-out)
Data Warehouse(MPP)
The Next Decade in Data Management
A stable system capable of variety of apps is necessary.Innovations in database are a requirement.New data stores are necessary.Differentiation between programs ill continue until key innovations in data management platforms become uniform.
OUR BIG-DATA/CLOUD PRODUCTS AND SOLUTIONS
PART-3:
Engineering
Overview
Cloud Solutions
• 以 Cloud Stack 云技术产品和方案为基础；
• 提供面向大规模数据存储和处理的行业应用解决方案 : Cloud Solutions ；
• 提供面向公众和企业的存储、计算、应用云服务产品 : Cloud Services ；
• 提供云应用 : Cloud Applications 。
电力
电信
交通
视频
物联网
互联网
医疗
政府 …
Cloud Applications(MagicBox, EnterpriseApps …)
Cloud Stack(CloudOS, SandStor, PebStor, MapReduce, vCompute, …)
Cloud Datasets
科研
NGO
…
Cloud Services (web-based)(ObjectStorage Service, DataStore Service,MapReduce Service, Compute Service …)
RESTful
Our Focus
• Enterprise Big Data Management
• Leverage of the Cloud Tech. from Internet Backend
Hardware
传统服务器： IBM 小型机 (p5 570) 联系集群系统 ( 深腾7000G) 曙光集群系统 ( 曙光TC5000) SUN 服务器 …
传统存储系统： NAS 系统 SAN 系统 磁盘阵列
弱点： 昂贵、扩展难、限制多
• 普通标准 PC 服务器• 自带存储 ( 单点可 >10TB)• 易维护• 节点可替代• 集群扩展方便• 组网灵活• Cluster-Level Soft RAID
拒绝昂贵、难扩展、局限性多的小型机、硬件捆绑集群和 SAN/NAS 等存储设备。
采用标准的普通服务器硬件 (PC-Server) 和网络设备，采用大数集群软件平台构建灵活的集群系统。集群规模可从几个节点到几千节点，存储规模可高达 PB 级。We rely more on software layer scalability (scale-out) and fault-tolerance.
Products and Features
April 10, 2023 17
CloudOS
SandStor PebStor MapReducevCompute
DataStore Cloud
ObjectStorage Cloud
MapReduce Cloud
Compute Cloud
Cloud API
Cloud Services
Cloud Stack
SandStor
• Distributed Structured Data Management
• Common features of CloudOS
• High efficiency Indexing
• Multi-level Cache• Compression• Fast random access,
Low Latency• Flexible Schema• High Durability, no
data loss
PebStor
• Distributed Blob Data Management
• Common features of CloudOS
• Efficiency indexes and meta mgmt
• Efficiency storage space mgmt
• De-duplicating• Unlimited blob size
MapReduce
• Flexible Parallel Data Processing Framework
• Common features of CloudOS
• Large-scale• High parallelized• Locality computing• Simple model for
programming• Abundant high-level
languages and toolkits
• Seamlessly integrated with storage system
vCompute
• Virtual Machines and Computing Resources mgmt
• Multi VMs support• Elastic VMs
provisioning• Auto-scale
CloudOS
• Distributed Cloud Platform• Commodity Hardware and
Cluster• High Scalability• High Reliability(Data Replication)• High Availability• Strong Consistency• High Throughput• Load Balancing• Global Data Access• Global File system• Simplify Complexity of Apps
Hardware & OS
Cloud Service Platform
• Multi-Level Cloud Services: – Infrastructure– Platform– Applications
Cloud Services 相似的同类产品或业务ObjectStorage Cloud Service Amazon S3
Google Storage for DeveloperRackspace Files/OpenStack SwiftGoogle BlobStore
DataStore Cloud Service Amazon SimpleDBGoogle DataStore
MapReduce Cloud Service Amazon MapReduceHadooop
Video Media Cloud Service … Video Delivery/Streaming/Transcoding/Time-shifting/Analytics
• Cloud Services API
– 基于 Web ，随处可得– RESTful 风格，简单易用– 提供对语言开发 SDK
• Cloud Services 的特点– 用户无需关心实现– 随处可得– 数据可靠性高– 伸缩性强– 可用性高 (99.9%)
– 按实际使用付费– 简单易用– API 符合业界标准 / 习惯– 丰富的管理和监控工具– 严密且灵活的安全策略– 多种云服务整合的 AAA 服
务
Object Storage Platformbuild another S3
RockStor Object Storage system provides object storage infrastructure services which guaranteed efficiency, robustness and load-balance.RockStor Object Storage system provides object storage infrastructure services which guaranteed efficiency, robustness and load-balance.
Data Chunk Store Layer Autonomous Overlay Network
Object Access Layer Providing Client Lib
MetaStore Layer DHT-based Consistent Overlay Network
Clustered storage nodes
Object-Oriented
High Availability
High Scalability
Huge Capacity
Object Storage Cloud Services
RESTful API 举例( 一个简单的对象上传 /PUT 操作 )
Object Storage Web-based 管理系统和 Amazon S3 类似
Performance of S3Write(8KB) Read(8KB)
Total Data size(GB) 1024 （ =1TB ） 1024 （ =1TB ）Total operations
count 134220800 134220800
Total used time(hour)
4.93 17.267
Total throughput/sec 7084.320 2155.119
Total average latency(us) 132.230 464.012
1306028040000 1306030200000 1306032360000 1306034520000 1306036680000 1306038840000 1306041000000 1306043160000 13060453200000
100020003000400050006000700080009000
10000dThrou(ops/sec)tThrou(ops/sec)
DataStore Platformbuild a scalable BDMS
API, SQL, Hadoop MapReduce 接口
BDMS 集群
应用层
集群服务层 集群节点网络拓扑 故障监测 分布式异步通讯框架
分布式存储平台层 分布式数据存储 负载均衡 数据副本和一致性管理 数据寻址
分布式存储引擎层 WAL ，写缓存和读缓存 存储文件结构和索引结构 数据压缩和压紧 数据分布管理和索引 本地分析引擎
数据模型和表述层 数据模型和 Schema 定义，存储引擎映射 索引管理 简单关系模型
数据访问层 SQL 语言， JDBC Driver API 导入工具 数据分析接口 ( 包括 Hadoop 集成接口 )
BDMS 软件层次架构BDMS 逻辑架构
Structured/Semi-
High Availability
High Scalability
Big Data
Performance of BDMS
1 21 41 61 81 1011211411611812012212412612813013213413613814014214414614815015215415615816016216416616817010
20000400006000080000
100000120000140000
write ops/Sec
totalThroughput deltaThroughput
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 330.00%
10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%
percentage of read ops
100ms
SLA of Random Query
Streaming Ingest Data Throughput
Query Result select * from table where msisdn > xxx limit N;
limit 1 0.34 secondlimit 10 0.31 secondlimit 100 0.40 secondlimit 1000 0.46 secondlimit 10000 1.25 secondslimit 500000 55.42 seconds
CloudNAS+MagicBox Enterprise Solution
• CloudNAS NAS Proxy + NAS in BigdataCloud
– File Server– Archive Server– Backup Server
Company LAN or WAN
Enterprise Private BigdataCloudWeb Service
RESTful API
Access files via CIFS/NFS/FTP
办公 /SOHO 网络
BigdataCloud NAS Proxy
MagicBox Service
MagicBox Client
• MagicBoxBackup/Sync/Sharing/Versioning– Documents Backup– Collaboration
Parallel Computing Platform
Data Split-1
Data Split-2
Data Split-3
Data Split-4
Data Split-5
Map-1
Map-2
Map-3
Map-4
Map-5
Reduce-1
Applications
job launch
assign map
Dataset as Input. Partition/Split as used
defined policy
Reduce-2
Output-1
Output-2
MapReduceJobTracker
MapReduce
BSP
Cloud Management
Thank You Very Much!Any more question?
http://cloudepr.blogspot.comhttp://www.slideshare.net/schubertzhang