polyglot persistence - confoo 2016

PolyglotPersistence

Majid Fatemian

R E D V E N T U R E S

@majidfn

Relational Non-Relational Key-Value Big Data

Application

Relational

Application

Relational Relational Non-Relational

Relational

BIGDATA

Capture

Search Storage

Q u e r y

Privacy Visualize

Analysis

Non-Relational

1 2 3Non-Relational Cluster Schemaless

2 x

1 x

1 x

2011

2012

2013

2014

2011

2012

2013

2014

Application

ACID

Normalized{ConsistencyRelational

Player ID Map ID Difficulty Score1000 223 E 52000 673 H 4.51000 451 M 42000 980 H 3

Activity

Player ID Name Location1000 John US2000 Catherine CA3000 Paul CA4000 Kylee US

Profile

Replication

Fault ToleranceAvailability{Relational

OPs / DBAs Tools Devs{DevOps

Relational

2011

2012

2013

2014

ScalabilityRelational

DistributionRelational

Data CData B

Data A

Data Access Layer

SchemaRelational

{ player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: medium, repetition: 178, score: 450, endurance:3}

{ player_id: 1234, level: 12, duration: 120, activity: activity}

Total

Detailed

stats

stats

Players’ Profiles

Individual Activities

Aggregatedstats

Categorized

ACID Transaction Partitioning Replication

Tools OPs

Data Volume Fixed Schema Scale Up Manual Sharding

+ -

NoSQL...

mongoDB

1 2 3OPs Devs Community

Schema-lessNon-Relational

player_id: 1234, * level: 40, * duration: 75, * activity: activity, game_id: 4001, platform: WiiU, difficulty_level: 2, repetition: 178, score: 450, endurance:3

player_id: 1234, level: 75, duration: 120, activity: activity

ScalabilityNon-Relational

ScalabilityNon-Relational

+

ShardingNon-Relational

Data A

{key}

ShardingNon-Relational

Data A

A

Data B

B

Data C

C

{key}

Map/ReduceNon-Relational

QueryK, V

K,V1 K,V2 K,V3

Map/ReduceNon-Relational M

ap

K,V1 K,V2 K,V3

Map/ReduceNon-Relational

ResultReduce

Map

K,V1 K,V2 K,V3

DenormalizedNon-Relationalread / }write

Disk Space

Player ID Map ID Difficulty Score1000 223 E 52000 673 H 4.51000 451 M 42000 980 H 3

Activity

Player ID Name Location1000 John US2000 Catherine CA3000 Paul CA4000 Kylee US

Profile

{Name: John, game:1, activity: Mission01, Score: 1245}

{Name: John, game:3, activity: Mission04, score: 5431}

{Name: John, game:1, activity: Mission01, Score: 1245}

Data ModelNon-RelationalQuery

Aggregation}

Eventual- Consistency

Non-Relational

X2

W

X1 X1

X2

Eventual- Consistency

Non-Relational

X2

W

X1 X1

R

X1

X2

Eventual- Consistency

Non-Relational

X2

W

X2

X2

R

X2

X2

BalancerNon-Relational

BalancerNon-Relational+

BalancerNon-Relational

Non-RelationalResilience

A B C

Non-RelationalResilience

A C

LockingNon-RelationalDB Level (2.2)}Document Level (3.x)

BackupNon-RelationalPoint-in-time

Denormalized data Disk Space Expertise Complex QueryingEventual Consistency Resource Usage DB-Level Locking

-

Schema-Less Aggregated Data Large Scale Data

Sharding Map/Reduce

Memory Storage Journaling

+

Random ProfileSearch(1M)

Full History Retrieval(6M)

Insert(6M)

AggregationMap/Reduce

(6M)

0

0.003

0.005

0.008

0.01

MySQLMongoDB

PersistencePolyglot

Players’ Profile Leaderboards

Activity Details

Relational Data

Schema-less Sharding

Total Sum

Activity 1

Activity n

Activity m

……

Shard 1

Shard 2

Config

Data Access Layer

Application

TheChallenge

Inconsistency1

Data Access Layer

W

Pending

Commit2-Phase

Data Access Layer

Pending

W Commit2-Phase

Data Access Layer

Done

Commit2-Phase

2 Latencyvs. Consistency

Operations3 Exponential complexity

Availability4 Plan for failure

Data Access Layer

Data Access Layer

Data Access Layer

Data Integrity5Backward compatibility

Unit{TEST IntegrationFunctional

Data Migration6

Profile

History }{ profile: { GUID:…, platform: { name:’xbox’, xuid:… } }, activity:…, duration:…, datetime:… }

3rd Party

1 One-Shot Migration4 hours !!

Live !!2 Profile versioning

V+11

V+11

V

+1

2

FinalResults

Scalability

2011 2012 2013 2014

9x response time

3 years uptime

Go Hybrid!Thoughtfully

>

ThankYOU!@majidfn

References

Polyglot PersistenceNoSQL Distilled