track1 02. 라쿠텐 트라벨 next search platform구축까지의 이야기 그리고...

My working life in Rakuten (Shim)&Rakuten Travel : New Search Platform (Jun)

Vol.01 2016-11-25Shim Jaemin, Jun Minsoo

Rakuten Inc.

Agenda

2

Introduction of our organization

My working life in Rakuten

Verifications

DataDistributionandSync

Summary

Agenda

3



Verifications


Summary

4

OEAA

Department

Marketing&MediaDevelopmentDepartment

1.MediaPropertiesSection

2.ResearchServiceSection

3.MarketingPlatformSection

OEAA

Marketing&MediaDevelopmentDepartment

3.MarketingPlatformSection

Marketing Platform Section

l We develop and operate advertising and marketing services using RaktenIchiba and internal media as well as external media.

l Mission： TodrivemarketingservicesofallRakuten.

Marketing Platform SectionJapan

Merchant AdDeliver &Analytics

RakutenAffliliate

Merchant Ads

Data Management

External AdsASB

External AdsRMJ

Rebates

Mail Service

Ad Solution Mail Service ArchitectureCommittee

RakutenAffiliateMerchant Ads

There are many global group and global servicesOffering variety of Internet marketing services.

There are many global group and global services

50people

25people

• 120people• OfficialLanguage:English

There are many global group and global services

Some typical services

Advertisers Publishers

• DisplayedonIchibaWebSiteandMail(+someofRakutenGroupMedia)• Cost-per-durationbyAdFramemodel• Adrevenue:2.5BNYen (2015-Feb).60% ofTotalMerchantAdrevenue.

Saleschannels Consumers

Ichibamerchants(14%)

(Top/Genre/Search/Event/Superdeal,etc…)

Inside ofIchiba

ECC

TenancyDisplayAds

Mail

CostperdurationtoAdFrames

Trackclickandpurchasethroughadsinternally.



• DisplayedonInsideofIchiba,OutofR-Group,Mail• Paid-per-clickmodel,SearchKeywordMatchandTargetingMatch• Adrevenue:0.51BNyen (2015-Feb).15% oftotalmerchantads.YoY+94%.


Ichibamerchants(18%ofall) (Throughad-networks)

Inside ofIchiba

Click!

ECC

Convertclicktorewardwhen

consumersclick anadvertising.

CPCAdOut ofR-group

Mail



• AllmerchantsareautomaticallysubscribedtoRakutenAffiliate.• Bigtouchpoints!GMSthroughR-Affiliateis65BYen/month


All Ichibamerchant

+Group

merchants

(Ex.Kakaku,Sansupo,Asahi,Reutersand

bloggers)

Out ofR-group $$$

ECC Convertclicktorewardwhenconsumers

purchase anitem

RakutenAffiliate

Purchase



• CoveringvariouskindsofdifferentMarketingSolutionservicesandDisplayadvertisingservices.- “AdbusinessforExternalClientsutilizingRakutenUsers.”-


ExternalClients

(Ex.Media Biz)

Inside ofR-group

RMJSales

Outside ofR-group

(Ex.Ad-network,DSP,Exchange)

Rakuten userCampaignfee

(Touchpoint:Mail,Point,Banneretc)

NationalClients(Ex.Nissan)

Externalagency

DSP CRM /LGNS

Amateras(Point campaign system)

DBM PointGallery

R-Calendar


SearchEngineResultPage(SERP)

SmartphoneApp(ByRewardBiz)

iOS Android

WebSearch

GetKuchi-su!

RewardSDK

Ø RakutenUsercangetincentive(Kuchi-su)orSuperPointbyusingRewardApp!

AchieveAction

GetSuperPoint!

SmartphoneApp(byOtherBiz)

Achieve

Actio

n

Features of development / operations team

ORACLE

SPRING

Agenda

15



Verifications


Summary

Where am I working?

Marketing Platform SectionJapan

Merchant AdDeliver &Analytics

RakutenAffliliate

Merchant Ads

Data Management

External AdsASB

External AdsRMJ

Rebates

Mail Service

Ad Solution Mail Service ArchitectureCommittee

RakutenAffiliateMerchant Ads

I’mhere!

Ø IamworkingforArchitectureCommitteeGroupwhichishorizontalorganizationofMPS

Organization of Architecture Committee

System#1PIC System#N PIC

AdditionalPost:20

Mainpost:2

…

HardWare

OperatingSystem

MiddleWare

Application

Shim,Jaemin|Daniel|MPDInformationEngineeringMaster15yearsofdevelopmentcareerSoundknowledgeofLinuxOSKernelandfamousmiddlewares.

Kim,Jinyong|Atssa|MPD.

Mission of Architecture Committee

ServicePlanning(BU)

SystemImplementationSystemOperation(MPS)

BetterSystemImplementation(ArchitectureCommittee)

Service

System

Architecture

WeareimprovingsystemqualityviaTECH.andbelieveinthefactthatitfinallyleadstohigherservicequality,either.

Activities of Architecture CommitteeA. Tech Information Sharing

- New Tech Introduction / Evaluation

- Explore Current Tech Already Used And Share

B. Standardization- Procedural Workflow Common To All MPS

- Common Tool Management For All MPS

C. Tech Support- Low Layer Tech Support Such As Hardware, Middleware, OS, etc.

- Tech Support for the problem beyond specific team’s capacity.

D. Architecture Review- It should assure high system quality by finding technical issues

- Providing countermeasures to them before development execution phase.

Working as an engineer at RakutenA. Global Company

B. Big system

C. Chance to growth

240 Gbps is the peak traffic during our Victory Sale*10% of Japan’s internet download traffic on that day

3.0 petabytes Datain Rakuten’s Hadoop

SupportemployeestoimprovetheirEnglish• OnlineEnglishlessons• FreeTOIECtests• Englishworkshopsandcircles

Engineers at Rakuten Inc, Japan• 45.8% of employees are non-Japanese• From 65 different countries across the globe

Agenda

21



Verifications


Summary

22

Who Are We

• Join to the Rakuten Travel : In 2006

• Search Platform Team Leader• Data Administrator Team Leader• DevOps : Automation, CI/CD

MinsooJunLeadArchitect/Rakuten Inc.

23

Rakuten Travel : New Search PlatformSince 1996 ~

28,000 JapanHotels+ GlobalHotels

110millionRakuten members

Sales 638billionYen

Our Issues

24

SearchPlatform

MasterDB

FrontServers

Onmemory

DB

SearchEngine

Onmemory

DB

SearchEngine

Onmemory

DB

SearchEngine

Onmemory

DB

SearchEngine

CURD

$$$$

Verifications

25

Lowlatency

HorizontalScale out

SLA99.95%

FaultTolerance

FlexibilityCostEfficiency

Sync withMasterData

Troubleshooting

Verifications

26

HorizontalScale out

SearchDB

MasterDB

SearchDB

SearchDB

SearchDB

SearchDB

SearchDB

SearchDB

SearchDB

Verifications

27

Lowlatency APIResponseTimeunder500ms

FrontServers BackendServers

SearchDB

Responsetime <500ms<2,000ms

Verifications

28

Troubleshooting Logs,Monitoring,Sources

Verifications

29

Service-levelagreement(Quality,Availability,Responsibilities)

SLA99.95%

Verifications

30

Canservicecontinuouslyevendoublefailure.FaultTolerance

Verifications

31

Infrastructure,Softwarearchitecture,Maintenance

CostEfficiency

Verifications

32

Caneasilyresponseaboutapplicationrequestwithoutbigschemachange

Flexibility

Verifications

33

AutomaticSynchronization,EventuallyConsistent

Sync withMasterData

Real-Time?<1sec

<1msec

<1nsec

?

Verifications

34

Lowlatency

HorizontalScale out

SLA99.95%

FaultTolerance

FlexibilityCostEfficiency

Sync withMasterData

Troubleshooting

DefineRequirements

PublicDocumentsCheck

PoC inCloud

Verifications

35

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency

FlexibilitySync withMasterData

*In2013

Verifications

36

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

37

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

38

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

39

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

40

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

41

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

42

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

43

Cassandra

OracleCoherence

MemSQL

MongoDB ActiveMQ

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

Verifications

44

MongoDB

RabbitMQ

Redis

MessagePack

JavaCache

HorizontalScale out

Lowlatency

Troubleshooting

SLA99.95%

FaultTolerance

CostEfficiency


* In2013

BuildtheSyncSystem

HowwecanSync?

Agenda

45



Verifications


Summary

Data Distribution and Sync

46

MasterDB

28,000JapanHotels

Global OTADirect 110millionRakutenmembers

Sales6,38billionYen

Rakuten Travel

RealTimeBooking

Read25vsWrite1


47

MongoDB

Redis

SearchEngine

MasterDB

DynamicData

StaticData

Front

MessagePack

Search

?


48

MongoDB

Redis

MasterDB

DataSync

InventoryAPI

Producer

RabbitMQ Consumer

DynamicData

StaticData

43


49

MongoDB

Redis

MasterDB

DataSync

InventoryAPI

Producer

RabbitMQ Consumer

DynamicData

StaticData

43

IndependentApplications

>BashShell : BatchScripting

DirectDMLissue

Queue orderissues

Interferenceproblems


50

MongoDB

Redis

MasterDB

DataSync

InventoryAPI

Producer

RabbitMQ Consumer

DynamicData

StaticData

43

IndependentApplications

>BashShell : BatchScripting

DirectDMLissue

Queueorderissues

Interferenceproblems

API-lization andapplyProducer

MakeaWEBUI

IssuetheProducerintheShell

SendtoonlyKeyvaluesforre-sync


51

MongoDB

Redis

MasterDB

DataSync

InventoryAPI

Producer

RabbitMQ Consumer

DynamicData

StaticData

43

500 millionQueues/Daily


52

MongoDB

Redis

MasterDB

MultiDC

InventoryAPI

Producer

RabbitMQ Consumer

DynamicDataStaticData

43

MongoDB

RedisRabbitMQ Consumer

Federate-amp-directDC1

DC2

Agenda

53



Verifications


Summary

Result : Architecture

Result : Mongodb

Result : Redis

TSP (Travel Search Platform)

ProvidesreliableNoSQL-basedsearch

99.98%guaranteedataSyncwithMasterDB

RemovingrepetitivetasksinAutomation

Supports11languages

LogPlatform :Foralllogtracking

APMManagement

ReleaseServiceincludebuildCI/CDpipeline

58

Come to our booth

59

Join ushttp://global.rakuten.com/corp/careers/engineering/

[email protected]

https://www.facebook.com/shim0413

[email protected]

https://www.facebook.com/minsoo.jun