big data : the missing puzzle of mobile computing

PowerPoint Presentation

From WSN to Mobile Computing, Big Data : The Missing PuzzleJazz WangYao-Tsung [email protected]

WHO AM I ? JAZZ

/

[email protected]

http://trac.nchc.org.tw/cloud

FOSSDebian/UbutnuAccess GridMotion/VLCRed5Debian RouterDRBL/ClonezillaHadoop

DRBL/ClonezillaPartclone/TuxbootHadoop Ecosystem

TRTC WSU/Haduzilla / Hadop4Win / Ezilla

DECLARATION

I'm NOT Export of Mobile Computing.My current research topics are:Cloud Computing and Big Data.I worked on few WSN projects years ago.In this talk, I'm trying to sharethe magic journey of sensor data to mobile computing.

Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ? Conclusion

This is a story about Data Flow

SensorNetwork

SmartGrid

Magic Journey of Data

Internet of Things

Open Data

Big Data

Mobile Computing

Cloud Computing

HISTORY

(Wireless Sensor Network)

SensorNetwork

Camera

InfraredLight

Photo-meter

Thermo-meter

Hygro-meter

WaterGauge

Speaker

Micro-phone

Video

Audio

Data

(Ecology Grid)

http://ecogrid.nchc.org.tw

1st camera

2nd camera

3rd camera

Optical connector

Field server

Taipower Admin office

KenTing Ecosite Geography Topology

EcoGrid Website PC

HiNet ADSL

512K/512K x 2

KenTing Ecosite PC

MPEG-1

Disk

Disk

Tape

LAN

Current Usage:90G @ 2004/09/01

NCHC KM Storage

KenTing Dataflow and Storage

Underwater Camera

Site A

Site B & C

Site B & C

(Agriculture Grid)

http://sensor.nchc.org.tw/tounan

AP Client

DataLogger

Sensor

Field Server

Pigeon House

AccessPoint

VideoServer

Camera

Router

ADSLmoderm

AP Client

DataLogger

Sensor

AccessPoint

VideoServer

Camera

Router

Bridge

Bridge

TANET

Internet

Special Education School

NCHC

Electric Wire Post

Electric Wire Post

Passive Field Server

(TRTC Neihu WSU)

http://www.flickr.com/photos/rail02000/5292653579/lightbox/

Wireless Fast Roaming

AP #1

AP #2

WSU

TRENDS

Internet of Things

Attacks on Mobile and Embedded Systems: Current TrendsbyMocana

http://www.techbang.com/posts/1258-wifi-scales-through-the-internet-to-manage-your-weight

Smart Grid

NIST

The Trends of Cloud Computing

Google Trends

200720112012

http://www.google.com/trends

Open Data

http://data.gov.uk/

Open Data

http://www.mygonews.com/news/detail/news_id/101779

Ocean Database

http://oceandb.info

Agenda From WSN to IoT? What is Cloud ?

WHAT

What is Cloud Computing ?

http://www.youtube.com/watch?v=bJLSAcU6O3U http://www.youtube.com/watch?v=VIMtd3nfPqc 8

Paradigm Shift of Cloud Business Model !!

Office 2007 Google Docs / Office 365

Outlook Webmail Mail Web Apps Mail Mobile Apps

PC / Server Hosting / Colocation Amazon EC2 / S3

PC / Server NB / Tablet Pad / Mobile

National Definition of Cloud ComputingNIST

5 Characteristics

4 Deployment Models

3 Service Models

1. On-demand self-service.

2. Broad network access

3. Resource pooling

4. Rapid elasticity

5. Measured Service

Enterprise iskey market

Private Cloud

4 Deployment Models of Cloud Computing

Public Cloud

Target Market is S.M.B.

HybridCloud

Dynamic Resource Provisioningbetween public and private cloud

Community Cloud

Academia

3 Service Models of Cloud Computing

SaaSSoftware as a Service

PaaSPlatform as a Service

IaaSInfrastructure as a Service

2 perspectives : Services vs Technologies?Cloud computing hype spurs confusion, Gartner sayshttp://www.computerworld.com/s/article/print/9115904 (Cloud Computing)http://www.cc.ntu.edu.tw/chinese/epaper/0008/20090320_8008.htm

One key spirit of Cloud Computing!!

Anytime

Anywhere

With Any Devices

Accessing Services

Cloud Computing =~ Network Computing =~

Key spirit of Cloud ~!!Everything as a Service !!

WHO

Supply Chain of Cloud Industry !!

\\

Who are the Cloud Service Providers ?

Enterprise iskey market

Private Cloud

Public Cloud

Target Market is S.M.B.

HybridCloud

Community Cloud

Academia

WHEN

Source: http://www.cnet.co.uk/i/c/blg/cat/software/cloudcomputing/clouds1.jpg

200689

GoogleEric SchmidtSES'06

Cloud Computing

2006824

AmazonElastic Compute Cloud

The Wisdom of Clouds (Crowds)

Mobile Cloud Service

Share Service Software

Personal Software

Physical

Mobile Mail

Web Mail

E-Mail

Mailbox

Mobile TV

Web TVEx. Youtube

Setop Box

TV

M-Office

Google Docs

Office

Typer Writer

Flash Wengo

Skype

PBX

Telephone

Twitter

Blog

BBS

Bullet Borad

Evolution of Cloud Services

WHY

Why are they named by SMART ?!

Smart Phone

Smart Grid

Smart Home

Smart Car

Smart City

Smart Meter

SMART ?

Wisdom

Knowledge

Data

Can Machine understand You?

http://www.ettoday.net/news/20120215/25085.htm

Data, Information, Knowledge, Wisdom

http://www.pursuantgroup.com/blog/tag/dikw-model/

Agenda From WSN to IoT? Cloud and Big Data What is Cloud ?

Key Driving Forces of Cloud Computing

Mobile Service

Cost Down

Data Explore

"It's the economy, stupid"James Carville42- 1992

"It's STILL the economy, stupid"- 2002

"It's the data, stupid"- 2007

Data Explosion!!2007

The Expanding Digital Universe, A Forecast of Worldwide Information Growth Through 2010,

March 2007, An IDC White Paper - sponsored by EMC

http://www.emc.com/collateral/analyst-reports/expanding-digital-idc-white-paper.pdf

2007IDC20102006

2006 161 EB

2010 988 EB ()

Extracting Value from Chaos,

June 2011, An IDC White Paper - sponsored by EMC

http://www.emc.com/collateral/about/news/idc-emc-digital-universe-2011-infographic.pdf

IDC

2006 161 EB

2007 281 EB2008 487 EB2009 800 EB (0.8 ZB)

2010 988 EB ()

2010 1200 EB (1.2 ZB)

2011 1773 EB ()

2011 1800 EB (1.8 ZB)

Data expanded 1.6x each year !!1.6

TBPB'Big Data' = few dozen TeraBytes to PetaBytes in single data set.

What is Big Data?!

100TB

100TB

100TB

http://en.wikipedia.org/wiki/Big_data

Gartner Big Data Model ?

Volume (amount of data)

Velocity (speed of data in/out)

Variety (data types, sources)

Batch ()

Realtime ()

TB

EB

Unstructured

Semi-structured

Structured

PB

[1] Laney, Douglas. "3D Data Management: Controlling Data Volume, Velocity and Variety" (6 February 2001)[2] Gartner Says Solving 'Big Data' Challenge Involves More Than Just Managing Volumes of Data, June 2011

12D of Information Management? 12

Source: Gartner (March 2011), 'Big Data' Is Only the Beginning of Extreme Information Management, 7 April 2011, http://www.gartner.com/id=1622715

Big Data

HOW

Devices share the wisdom of Cloud

New IT Architecture toward Cloud Computing !!

1. 6. (

2.

3.

4.

5.

Three Key Technologies !! vs.

IaaS

Infrastructure as a Service

PaaS

Platform as a Service

SaaS

Software as a Service

Virtualization

Big Data

Web 2.0

Mobile Service

Cost Down

Data Explore

Agenda From WSN to IoT? Cloud and Big Data Smart City AppWhat is Cloud ?

http://www.postscapes.com/anatomy-of-a-smart-city-full


200850%



2011





6D of Smart Cities

SmartLivingSmartGovernanceSmartPeopleSmartEconomySmartEnvironmentSmartMobilitySustainablehttp://en.wikipedia.org/wiki/Smart_city

EducationTrafficHousingFoodEntertainmentClothing

Ubiquitous Computing

http://www.ipeen.com.tw/map/#loc=

http://www.cwb.gov.tw/township/

http://rent.591.com.tw/

http://1968.freeway.gov.tw/

http://www.datanami.com/datanami/2012-08-01/what_it_takes_to_deliver_real-time_traffic_info.html

http://hypercities.ats.ucla.edu/

http://earth.wra.gov.tw/water/index.html

Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ?

The SMAQ stack for big data

The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.htmlhttp://smashingweb.ge6.org/wp-content/uploads/2011/10/apache-php-mysql-ubuntu.png

LAMP

SMAQStorage, MapReduce and Query


The SMAQ stack for big dataEdd Dumbill22 September 2010http://radar.oreilly.com/2010/09/the-smaq-stack-for-big-data.html





Three Core Technologies of Google ....
Google ....

Google

Google shared their design of web-search engine

SOSP 2003 :

The Google File System

http://labs.google.com/papers/gfs.html

OSDI 2004 :

MapReduce : Simplifed Data Processing on Large Cluster

http://labs.google.com/papers/mapreduce.html

OSDI 2006 :

Bigtable: A Distributed Storage System for Structured Data

http://labs.google.com/papers/bigtable-osdi06.pdf

Open Source Mapping of Google Core Technologies
Google

Hadoop Distributed File System (HDFS)

Sector Distributed File System

Hadoop MapReduce API

Sphere MapReduce API, ...

HBase, Hypertable

Cassandra, ....

S = Storage

Google File System

To store petabytes of data

MapReduce

To parallel process data

Q = Query

BigTable

A huge key-value datastore

Google

Hadoophttp://hadoop.apache.org

HadoopApache Top Level

Hadoop is Apache Top Level Project

Yahoo!

Major sponsor is Yahoo!

Doug CuttingGoogle Filesystem

Developed by Doug Cutting, Reference from Google Filesystem

JavaHDFSMapReduce API

Written by Java, it provides HDFS and MapReduce API

2006Yahoo

Used in Yahoo since year 2006

It had been deploy to 4000+ nodes in Yahoo

Petabyte

Design to process dataset in Petabyte

FacebookLast.fmJoost are also powered by Hadoop

Hadoop

map reduce

HDFS

Map

Reduce

Hadoop

Sector / Spherehttp://sector.sourceforge.net/

Developed by National Center for Data Mining, USA

C/C++ Hadoop

Written by C/C++, so performance is better than Hadoop

Google File SystemMapReduce

Provide file system similar to Google File System and MapReduce API

UDT

Based on UDT which enhance the network performance

Open Cloud TestbedMalStone

Open Cloud Consortium provide Open Cloud Testbed and develop MalStone toolkit for benchmark

Why we choice Hadoop? Good Ecosystem!

http://rationalintelligence.com/wp_log/?p=104

Agenda From WSN to IoT? Cloud and Big Data Smart City AppHow to do in future What is Cloud ? Conclusion

IoT is the source of Big DataCloud Computing is the StoreBig Data is the key to Mobile ComputingSMAQ is the way to achieve Wisdom

SMAQ

Questions?Slides - http://trac.nchc.org.tw/cloudJazz WangYao-Tsung [email protected]

Column 1

User50

Promoter35

Developer15

big data : the missing puzzle of mobile computing

Technology