© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

베스핀글로벌이한주대표

2016.5.17

AWS를활용한 Big Data 실전배치사례

모니터링, IOT의 Big Data 배치및Data Visualization 사례

About HanJoo Lee

CEO & Cofounder , BESPIN GLOBAL

General Partner,

1998 Cofounded

IT Infra entrepreneur

Built and Operated 12 Datacenters100,000+ 서버운영

11 Countries with 12 Datacenters around the world: 1998년 시작

Australia Belgium Canada France Germany Korea Netherlands USA UK India Rumania

Sydney Antwerp Vancouver Paris Frankfurt Seoul Amsterdam Chicago London Mumbai Bucharest

Hannover Tampa / Austin

Fort Lauderdale

About Bespin Global

베스핀글로벌Managed Service Provider한국에서최초로 AWS MSP 인증받은회사한국에는나와있는 3개 MSP 인증중 2개가 Bespin

• Cloud 전략수립• Cloud Architecture• Cloud Migration• Cloud 운영• Hybrid IT Management• 한국과중국에서 140명

About Bespin Global

Cloud의길잡이

Data lifecycle in Cloud Infra

클라우드 인프라

데이터 애플리케이션

데이터 입력 가공 분석 활용

• IoT를 통해 세상의 모든

것들을 디지털 데이터

로 수집

• 빅데이터 기술을 통해

과거의 텍스트 중심의

정형 데이터뿐만 아니

라 동화상, 음성 등 비

정형 데이터까지 처리

• 대규모 데이터를 실시

간 분석해 결과를 추출

하고 의사결정 진행

• 디지털 골든크로스로서

인간의 두뇌 이상의 의

사결정과 작업을 컴퓨

터가 스스로 수행

머신러닝

빅데이터

IoT

Bespin Global

김성수 상무

Building Big Data Backend Using Native AWS Services

WhaTap

김성조 CTO

Building Big Data System Using Proprietary File System while Using AWS

N3N

김호민 대표

Building Big Data Visualization Using Splunk and AWS

Big Data for SaaS based Monitoring Services

Bespin Global: How to build and Operate Big Data System on AWS?

Avoid unplanned Downtime

Monitoring Process

Server

DBMS

Application

Monitoring Target

분석 DB

Basic Process

Advanced Process

(사후 처리)

Metric

Data 수집 및모니터링

Big DataProcessing

※ Ca사 모니터링 솔루션 UIM 적용

Threshold Alarm

사전 예방 활동

Scale up

Scale out

(Time Series, Correlation, …)

알람 or 장애 조치

Monitoring Dashboard

Building Big Data System on On-Premise

R Analysis( Time Series / Correlation )

Pre detection

DatabaseDistributed Messaging System

…..

M N1 N2 Nn

Cluster ComputingFramework

Yarn

…..

HDFS

Time Series Database

N2 Nn…..M N1

Analysis Platform

……

Data Sources

Application

Server

DBMS

Log

Age

nt

…

Applications

BI

Marketing

Advertisement

…

Limited Scalability

Internal IT Resources to Manage Cluster (Tuning and monitoring, etc…)

Upfront capital expense

NoSQL

…

사전 진단/예방

ProducerStream

Collector

ConsumerStream

Building Big Data System on AWS

Data Sources

Application

Server

DBMS

Log

Age

nt

…

ProducerStream

Kinesis

Streaming Data Platform

EMR AWS Elastic Search

Time Series Database

RDS

Pre detection

KCL

Kinesis S3 Connector

Consumer

S3Lambda

Archiving Data

EMR

Redshift

RDS

An

alysis (BI So

lutio

ns)

Application

Upsell Analysis

Marketing

Advertisement

…

Elastic and Highly Scalable

Don’t Manage Cluster (and AWS has tuned Services)

Easy to Use and Deploy to Multiple Locations

No upfront capital expense – Pay as you go

Cluster monitoring

Cluster monitoring

R Analysis( Time Series / Correlation )

R on EMR

Consumer

사전 진단/예방

On-Premise vs AWS

R Analysis( Time Series / Correlation )

Database

Streaming Data Platform

Kinesis

Time Series Database

Elastic Search

Easy to Use and Managed Console

RDS

Database

Complex to Use Managed Cost+

Cluster Managed Cost (Tuning and monitoring, etc…)+

Elastic Scalability

Don’t Managed Cluster (and AWS has tuned Services)

Upfront capital expense

Pay as you go

Distributed Messaging System

…..

M N1 N2 Nn

Cluster ComputingFramework

Yarn

…..

Time Series Database

N2 Nn…..M N1

Limited Scalability

EMR

R Analysis( Time Series / Correlation )

R on EMR

WhaTap: Running Cloud Based Monitoring Service on AWS

Monitoring Issue Changes

Testing Open Stable

Application Physical SystemNeeds

Unknown bottlenecks

Known issues

Monitoring Solutions

Needs

Testing Open Stable

APM(분석, 문제 해결, 실시간)

SMS(관제, 경고 이벤트)

The differences on Cloud

Dedicated Unix Infra

개별 서버 성능

서버 관리 유연성

서버(노드) 수

시스템 처리 성능

부분 오류 가능성

Cloud Infra

서비스 안정성

Monitoring Service(SaaS based)

Master

security group security

1

2

200 Slave

Monitoring Service

APM on Premise vs on Cloud

• Data collection per several seconds• Active Transaction Analysis• All Transaction Profiling

On Premise Solution

• Data collection per 10 seconds• Active Transaction Analysis• Selective Profiling & Integrative Analysis

On Cloud Service

사용자접속정보,

트랜잭션, 자원, 튜닝정보를

하나의관점으로비교분석할수

있어야합니다. Archiving

TechUser (browser)

Transaction

IP, 도시/국가,

접속 매체, OS,

최근 방문자,

액티브 사용자

TPS, Response

time, Error

time, URL, SQL

Resource

CPU, Heap,

Disk, GC

Tuning

History 비교분석

Active Stack 분석

Hit Map 분석

Integrated analysis of perf. data

EC2 instance

DataServer

EBS

Backup (S3)

StaticsticsRDS (MySql)

Scalable APM (AWS based)

EC2 instance

DataServer

EBS

EC2 instance

DataServer

EBS

EC2 instance

DataServer

EBS

Scalable!!

Elastic Load Balancing

Amazon Route 53

Auto Scaling group

EC2 instance

DataServer

EBS

Backup (S3)

Project/Tenant MgmtRDS (MySql)

StaticsticsRDS (MySql)

ElastiCache(REDIS)

Elastic Load Balancing

Amazon Route 53

Auto Scaling group

Region (Tokyo)

Elastic Load Balancing

Auto Scaling group

EC2 instance

DataServer

EBS

Backup (S3)

StaticsticsRDS (MySql)

Region (US.West)

ElastiCache(REDIS)

Elastic Load Balancing

Auto Scaling group

Region (Tokyo)

Region (US.West)

Elastic Search

Scalable APM (AWS based)

APM to SMS on Cloud

Needs

Testing Open Stable

APM• 분석, 문제 해결• 실시간• 대용량 데이터

SMS• 관제, 문제 인지• 경고 이벤트• 소규모 데이터

SUPPORTS

MOBILE

LOWER

COST

SHORTER

IMPLEMENTATION

PERIOD

SMS on Cloud

1 Month

EXISTING MONITORING

CLOUD MONITORING

5 Minutes

$10000 + α

EXISTING MONITORING

CLOUD MONITORING

$100/mon

EXISTING MONITORING

CLOUD MONITORING

In Office

Anywhere

WhaTap TokyoWhaTap Virginia

WhaTap Oregon

출처 : https://rctom.hbs.org/

WhaTap can service to global customers

Big Data Visualization on AWS

N3N: What We Do

N3N provides IOT Visualization for Fortune 500 Companies and Cities Governments.

N3N - Physical Visualization

Global

Asia

Korea

Seoul

Suwon

Z: Hierarchies

Daejeon

X: Dependencies Y: Relationships

N3N - Logical Visualization

Business Unit

Service

Application

X: Dependencies Y: Relationships

ApplicationData Bases

Z: Hierarchies

N3N - Business Impact

Quantitative BenefitsKPI Before After Result Remarks

MTTD, MTTR 2 days 1.5 days -0.5 day CISCO Stat, Splunk .conf 2014

Big Data Solution Usage Rate 5% 100% +95%

Improvements in Stability & availability

+25%

Reduction in Operational Cost +10%

25% 95% 25% 10%

Wrap Up

Visualization Big data for Biz Decision

Monitoring Log

IOT event Log

System Event Log

AWS event Log

VisualizedAnalysis On AWS

Wrap Up

- Big Data on AWS –Monitoring Use Case

- Not only for Monitoring

- Merchandise, Logistics, Turnover, Marketing, etc.

- Best Practice & Consulting from Bespin Global

WE ARE HIRING!!!!• KOREA• CHINA• JAPAN• USAPLEASE JOIN US

감사합니다!

HELPING YOU ADOPT CLOUD.