[h3 2012] cloud database service - hulahoop를 소개합니다
DESCRIPTION
H3 2012 발표자료 Cloud Database Service - Hulahoop를 소개합니다. -KTH 황지수TRANSCRIPT
![Page 1: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/1.jpg)
Hulahoop : Cloud Database Service
분산기술Lab.
황지수
![Page 2: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/2.jpg)
3/44
Session Preview
간단한 소개
개발 배경
구조와 특징
서비스로의 진화
적용 사례
![Page 3: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/3.jpg)
간단한 소개 Hulahoop가 어떤 것인지 맛보기
![Page 4: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/4.jpg)
5/44
한 마디로 말해보기
Hulahoop는
이다 Cloud Database Service
![Page 5: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/5.jpg)
6/44
Cloud Database Service Taxanomy
• Cloud Database Service 분류
Virtual Machine Deployment
Database as a Service
SQL
Oracle
MySQL
PostgreSQL
Amazon RDS
Microsoft Azure
Heroku
NoSQL
CouchDB (EC2)
Hadoop(EC2)
Cassandra(EC2)
Amazon DynamoDB
Amazon SimpleDB
Google AppEngine Datastore
![Page 6: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/6.jpg)
7/44
Deployment Model
Virtual Machine Deployment
Database as a Service
설치 & 운영 사용 요금 지불
![Page 7: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/7.jpg)
8/44
Data Model
SQL NoSQL
SELECT [name]
FROM [H3_ATTENDEES]
WHERE [current_session] =
‘Hulahoop’
Set( key, value )
Get( key )
Delete( key )
표준 복잡한 모델
비표준 간단한 모델
![Page 8: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/8.jpg)
9/44
Hulahoop를 분류해보면,
Virtual Machine Deployment
Database as a Service
SQL MySQL Heroku
NoSQL Cassandra(EC2) DynamoDB, Hulahoop
![Page 9: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/9.jpg)
개발 배경 Hulahoop : The Beginning
![Page 10: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/10.jpg)
11/44
결론 먼저,
1. Cloud Database Service가 되기 위해서 만들어진 것이 아닙니다.
2. 분산파일시스템을 위해서 태어났습니다.
![Page 11: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/11.jpg)
12/44
분산파일시스템 구조
분산파일시스템 구조
Master Node
Data Node #2
Data Node#1
Data Node #3
Data Node #4
1,3,4
2,3,4
![Page 12: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/12.jpg)
13/44
이런 구조의 단점은,
구조적 한계
• SPOF
• Scalability
Master Node
Data Node #2
Data Node#1
Data Node #3
Data Node #4
1,3,4
2,3,4
![Page 13: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/13.jpg)
14/44
PrismFS v1.0
PrismFS v1.0
• Multi Master
• Cassandra
Data Node #2
Data Node#1
Data Node #3
Data Node #4
확장성 + 가용성
Master Node
Storage Storage
Cassandra
![Page 14: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/14.jpg)
15/44
Cassandra를 선택한 것은,
v0.6
고가용성 ★
수평적 확장성 ★
적용사례 ★★★
데이터 모델 ★★★
고가용성 ★★★
수평적 확장성 ★★★
적용사례 ★
데이터 모델 ★
![Page 15: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/15.jpg)
16/44
Cassandra 적용 이슈 1. NoSQL 데이터 모델
Data Model
• De-normalization Table( c1, … , cn ) ( key, value )
• Data Marshalling
File Meta Slice Meta
Slice Meta
. . .
Marshalled Data
Key
![Page 16: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/16.jpg)
17/44
Cassandra 적용 이슈 2. Update pattern
Update Pattern
UPDATE [Meta] SET [version] = [version] + 1 WHERE [key]=‘key’
BLOB = get( key )
meta = UnMarsh( BLOB )
meta.version += 1
BLOB_v2 = Marsh( meta )
Set( key, BLOB_v2 )
Get & set
![Page 17: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/17.jpg)
18/44
Slice Meta
Slice Meta”
. . . Slice Meta’
Slice Meta
. . .
Cassandra 적용 이슈3 . Concurrent update
Concurrent Update
• Locking ( Local / Global )
Slice Meta
Slice Meta
. . .
Node A get & set
Node B get & set
![Page 18: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/18.jpg)
19/44
Cassandra 적용 이슈4 . Timestamp ordering
Timestamp Ordering
100 110 120 Node A
20 40 50 Node B
100 50
120+α
>
<
100+α 110+α 120+α Logical Clock
![Page 19: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/19.jpg)
20/44
PrismFS v1.0에서 얻은 경험은,
데이터모델의 차이를 극복하려면,
• 구현 복잡도 증가.
• 메타데이타 저장소의 성능 활용 못함.
PrismFS에서 필요한 데이터모델
![Page 20: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/20.jpg)
21/44
새로운 메타 데이터 저장소
PrismFS v2.0을 위한 새로운 메타 저장소
• 고가용성
• 수평적 확장성
• 분산파일시스템에 적합한 데이터 모델
• 부분적인 ACID
Hulahoop
![Page 21: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/21.jpg)
구조와 특징
![Page 22: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/22.jpg)
23/44
Architecture Overview
Architecture Overview
• 데이터 모델
• Slot
• Slot Ownership
• Fail-over
• Scale-out
![Page 23: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/23.jpg)
24/44
Data Model
데이터 모델
기본은 간단한 데이터 모델
• ( Key, Value ) = ( Domain, Data )
다양한 Data 처리 방법으로 기능 확장
• SQL
• Python Script
Domain Data
![Page 24: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/24.jpg)
25/44
Data Type : SQL
Data Type : SQL
Hulahoop
Col1 … ColN
Row1_col1 … Row1_colN
… … …
RowN_col1 … RowN_colN
SELECT * FROM MY_TABLE;
![Page 25: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/25.jpg)
26/44
Data Type : Python Script
Data Type : Python Script
• Entry point.
• Server context.
Hulahoop def run( conn ): c = conn.cursor() c.execute( ‘…..’ ) return c.fetchone()
result
![Page 26: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/26.jpg)
27/44
Slot
Slot
• 데이터 처리와 저장 모듈
Operation Log
Database Engine
Script Processor
Snapshot
Data
Background
![Page 27: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/27.jpg)
28/44
Hulahoop와 Slot
Hulahoop = 다수의 Slot
Slot
Slot Slot
Slot
Slot Slot
Slot
![Page 28: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/28.jpg)
29/44
Slot Ring
Slot Ring
• 가상의 원형 공간에 배치
• 고정된 Ring의 크기 = Slot의 개수
Slot
Slot Slot
Slot
Slot Slot
Slot
Hash
Domain
![Page 29: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/29.jpg)
30/44
Server Ring
Server Ring
• Slot Ring과 같은 크기.
• 동일한 규칙으로 ownership 영역을 갖음.
![Page 30: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/30.jpg)
31/44
Slot Ownership
Slot Ownership
Slot
Slot Slot
Slot
Slot Slot
Slot
![Page 31: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/31.jpg)
32/44
Slot Routing
Slot Routing
Hash
Domain
![Page 32: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/32.jpg)
33/44
Replication
Replication
• Operation Log를 3배수로 복제.
![Page 33: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/33.jpg)
34/44
Failover
Fail-over
• 이웃한 서버에 대해서 작동.
Slot
Slot
Slot
Slot
Slot
Slot
![Page 34: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/34.jpg)
35/44
Failover
Fail-over
• 이웃한 서버에 대해서 작동.
Slot
Slot
Slot
Slot Slot
Slot
Activate
![Page 35: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/35.jpg)
36/44
Scale-out
Scale-out
• 새로운 Ownership에 따라서 Slot 재배치.
Slot Slot
Slot
Slot Slot
Slot
![Page 36: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/36.jpg)
37/44
Scale-out
Scale-out
• 새로운 Ownership에 따라서 Slot 재배치.
Slot
Slot
Slot
Slot
Slot
Slot
Slot Slot
Copy Data &
Acquire Ownership
![Page 37: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/37.jpg)
38/44
Features
Hulahoop 구조의 특징
• Decentralized system.
• Horizontal Scalable.
• Support SQL, Python Script.
![Page 38: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/38.jpg)
서비스로의 진화
![Page 39: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/39.jpg)
40/44
Amazon DynamoDB API
Amazon DynamoDB API
• AWS APIs 사실상의 표준
• PrismFS = AWS S3
• 다수의 3rd party tools, libraries, applications
• 풍부한 documentation
![Page 40: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/40.jpg)
41/44
Service Architecture
Service Architecture
RPC Server REST Server
Hulahoop
![Page 41: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/41.jpg)
42/44
Hulahoop, database service
Hulahoop, Database Service
사용자가 얻는 혜택
• NO, DBMS 설치&운영
• AWS DynamoDB의 장점
![Page 42: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/42.jpg)
적용사례 어떤 곳에서 쓰이고 있는지
![Page 43: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/43.jpg)
44/44
현재 : 적용사례
1. PrismFS 2.0 ( Distributed File System ) - AWS S3 API. - baas.io Storage Service. - Hulahoop Native API. - File metadata storage.
2. IRIS ( Cloud Search Engine ) - Hulahoop REST API. - Document storage.
![Page 44: [H3 2012] Cloud Database Service - Hulahoop를 소개합니다](https://reader033.vdocuments.pub/reader033/viewer/2022050808/54b4c1f54a7959bb058b4618/html5/thumbnails/44.jpg)
기술연구소 / 분산기술Lab.
황지수
jsioux {at} kthcorp . com