accelerating adtech on aws in japan

Accelerating AdTech on AWS in JapanPragmatic use-cases Dynalyst / fluct / IM-DMP / UNICORN

Eiji ShinoharaAmazon Web Services Japan, Solutions Architect

August 30, 2017 at MARU180

안녕하십니까!

@werner: Amazon CTO

Piljoong

Park-san

Eiji Shinohara (篠原英治)§ Twitter: @shinodogg§ Blog: shinodogg.comAWS Solutions Architect§ Market: AdTech & Startup§ Area of Depth: SearchKorean Cuisine Lover§ 三겹살/삼겹살, 불고기, 김치,,,

Agenda

Japanese AdTech IndustryJapanese AdTech CommunityAdTech on AWS use-cases in JapanvDynalyst http://www.dynalyst.io

vfluct https://fluct.jp

vIM-DMP https://corp.intimatemerger.com

vUNICORN https://uncn.jp

Japanese AdTech Industry

Japanese AdTech IndustryJP 2016 Internet Ads Market Size Research by CCI

http://www.cci.co.jp/news/release/2017_04_17/1.html

Japanese AdTech IndustryJP 2016 Internet Ads Market Size Research by CCI

http://www.cci.co.jp/news/release/2017_04_17/1.html

$10 Billion Market

Smartphone Shift

SmartphoneDesktop


https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016


https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016

Fragmented and Competitive same as other regions

Japanese AdTech Community

Japanese AdTech CommunityAdTech Meetup by AWS in 2016 #AWSAdTechJP

Japanese AdTech CommunityAdTech Meetup by AWS in 2016 #AWSAdTechJP

“Digital Marketing”Trend

DialogOne“LINE” Business Connect

“AdNetwork”Admin Tools

http://aws.typepad.com/sajp/2016/07/aws-adtech-jp.html

Wrap-up Blog Post

Japanese AdTech CommunityAkiba Lab – Over 800 people in Facebook group

Japanese AdTech CommunityAkiba Lab – Over 800 people in Facebook group

アドテク⇒AdTech

Akiba Lab is a Japanese AdTech communityBig year-end party in Dec 2016

Lightning Talks

Japanese AdTech Community#CTONight powered by AWS


Japanese Top-NotchCTOs gathering


v AdTech CTOs at CTO Night J

Japanese AdTech CommunityAdTech CTOs Round Table

v Deeper AdTech Talks!

Japanese AdTech on AWSUse-Cases

AdTech on AWS Use-Cases in JapanDynalyst http://www.dynalyst.io

v Re-Targeting / Re-Engagingv Japan and U.S.

fluct https://fluct.jp

v SSP: 30 billion impressions in a monthv Ajitofm: Podcast @ VOYAGE GROUP in company bar

IM-DMP https://corp.intimatemerger.com

v Public DMPv Small Engineering Team delivers Big Result

UNICORN https://uncn.jp

v Full Automated Marketing Platformv International Engineers in Tokyo

CyberAgent - Internet Conglomerate

https://www.cyberagent.co.jp/en/


http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf


http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf

About 500 million USDAd Business

CyberAgent - AdTech Studio

https://adtech.cyberagent.io/ https://www.slideshare.net/InsightTechnology/dbts-tokyo-2016-b15-bigdata-actian-matrix-mapr-cyberagent

Dynalyst - Dynamic Retargeting for Game Apps

Massive Audience

Personalized Engagement

AWS Summit Tokyo 2014


Dynalyst - Log Processing Architecture

Japan

US

ap-northeast-1

us-east-1

Up to100 instances

Up to80 shards

KCL on ECSDocker Cluster

S3

Redshift

EMR

Up to100 instances

Up to80 shards



Japan

US

ap-northeast-1

us-east-1

Up to100 instances

Up to80 shards


S3

Redshift

EMR

Up to100 instances

Up to80 shards



Petabyte Scale

Dynalyst - Cross Region ReplicationReal-Time Bidding

https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf

Dynalyst - Cross Region ReplicationReal-Time Bidding

https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf

Network Latency is Critical

https://www.mopub.com/resources/mopub-demand/mopub-marketplace-overview/network-infrastructure/

Dynalyst - Cross Region Replication

DynamoDB Streams

Amazon RDS Aurora

Dynalyst - Real-Time BiddingTrain Model: Spark ML / Save Model: Redis

Quick Response to Bid Requests!EMR ElastiCacheS3

Bid Request

Memcached

Redis Aurora

DynamoDB

Dynalyst - Go Global with AWS!Shuhei Kimura

v Moving back and forth from Japan to U.S.v Diving deeply into U.S. AdTech eco-systemv Planning to use another AWS region in US West

VOYAGE GROUP - fluct: SSP

fluct - Serverless Architecture in 2016Serverless for Analyzing contentsvBetter Contents/Context matched Ad delivery

https://speakerdeck.com/suzuken/how-to-use-aws-lambda-in-document-processing-pipeline

fluct – SSP: 30billion impressions in a month

Kenta Suzuki

A. Advertising transparency

v Players are relying on each other

v Preventing unethical actions is an entire industry problem!

v Letʼs make the Internet better place J

Q. What is the trend in AdTechindustry?

fluct magazine https://magazine.fluct.jp



“ads.txt” aims to increase transparency in the AdTech ecosystem

How do we introduce ads.txt?

fluct magazine https://magazine.fluct.jp

fluct - Ad Verification Architecture

Adtag

ALB ECS

Lambda

Lambda

Amazon ES

3rd Partyservice

Kinesis


Adtag

ALB ECS

Lambda

Lambda

Amazon ES

3rd Partyservice

Kinesis

Speed Layer

Batch Layer


Adtag

ALB ECS

Lambda

Lambda

Amazon ES

3rd Partyservice

Kinesis

Running “Golang” applicationon AWS Lambda w/ Apex

fluct - Serverless Architecture

Apex – Serverless Infrastructure“build, deploy, and manage AWS Lambda functions with ease”

http://apex.run/

https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex

Apex – Serverless Infrastructure

Running Golang on AWS Lambda

https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex

Running Golang on AWS Lambda

Apex – Serverless Infrastructure

Tech Podcast - VOYAGE GROUP

https://www.instagram.com/p/BXWzZ9ngYLZ/HUGO (https://gohugo.io/) + Hosting on S3

Ajitofm https://ajito.fm/

https://ajito.fm/2/

Tech Podcast - VOYAGE GROUPRunning Golang on AWS Lambda

v Node.js -> GolangRunning Golang as a Child ProcessUtilize STDIN and STDOUTSounds like “CGI” in Cloud ERA...

http://www.kent-web.com/

https://ajito.fm/2/

v Node.js -> GolangRunning Golang as a Child ProcessUtilize STDIN and STDOUTSounds like “CGI” in Cloud ERA...

Popular CGI ExamplesIn 90s…

Tech Podcast - VOYAGE GROUPRunning Golang on AWS Lambda

http://www.kent-web.com/

AJITO – VOYAGE GROUP Inside Company Bar

Intimate Merger - IM-DMP

Intimate Merger - IM-DMP

Intimate Mergerv Founded in 2013 as a Joint Venture

FreakOut: The first DSP in JapanPreferred Infrastructure: Cutting Edge Tech

v Shareholders in 2017FreakOut Holdings: Global Marketing Tech groupDentsu: Worldʼs leading Advertising AgencyYJCapital: Yahoo! Japan Corporate Venture Capital

Intimate Merger - IM-DMPw/ dentsu

v Contribute to Public DMP ”dPublic” by dentsuw/ Yahoo! Japan

v Connect to Yahoo! Japan DMP

https://corp.intimatemerger.com/archives/1855/

Intimate Merger - IM-DMPw/ dentsu

v Contribute to Public DMP ”dPublic” by dentsuw/ Yahoo! Japan

v Connect to Yahoo! Japan DMP

https://corp.intimatemerger.com/archives/1855/PsychographicDemographic

400 millionAudience Data

IM-DMP - Architecture

IM-DMP - Architecture

ECS

RDS

ElastiCache

Spot Fleet

ECS

CSVTSVJSON

SQS

IM-DMP

UI&API Data Processing

S3

3rd Party Partners

ECS

RDS

ElastiCache

CSVTSVJSON

SQS

IM-DMP


S3

3rd Party Data Providers

Everything is started from S3 uploadSimple File-based Trigger

Spot Fleet

ECS

IM-DMP - S3 File-based Architecture

ECS

RDS

ElastiCache

CSVTSVJSON

SQS

IM-DMP


S3

3rd Party Partners

ECS & Spot Fleet

Spot Fleet

ECS

IM-DMP - Amazon ECS & EC2 Spot Fleet


https://www.youtube.com/watch?v=vJZJY_t8JDE https://www.slideshare.net/im_docs/amazon-ecs-spotfleet

IM-DMP - Amazon ECS & EC2 Spot Fleet

Amazon EC2 Spot Instances

July 26, 2017 / ap-northeast-1 / LinuxOn

Demand

Reserved Instances for 1 yearSpot

Instances

Spot Block

All Upfront

Partial Upfront

No Upfront 1h 6h

c4.large $0.126 $0.084(33%)

$0.086(32%)

$0.090(29%)

$0.029(77%)

$0.077(39%)

$0.098(22%)

m4.large $0.129 $0.081(37%)

$0.083(36%)

$0.087(32%)

$0.027(79%)

$0.101(21%)

$0.128(0.7%)

r3.large $0.20 $0.127(36%)

$0.130(35%)

$0.149(26%)

$0.031(84%)

$0.116(42%)

$0.147(26%)

On Demand

Reserved Instances for 1 yearSpot

Instances

Spot Block

All Upfront

Partial Upfront

No Upfront 1h 6h

c4.large $0.126 $0.084(33%)

$0.086(32%)

$0.090(29%)

$0.029(77%)

$0.077(39%)

$0.098(22%)

m4.large $0.129 $0.081(37%)

$0.083(36%)

$0.087(32%)

$0.027(79%)

$0.101(21%)

$0.128(0.7%)

r3.large $0.20 $0.127(36%)

$0.130(35%)

$0.149(26%)

$0.031(84%)

$0.116(42%)

$0.147(26%)

July 26, 2017 / ap-northeast-1 / Linux

Amazon EC2 Spot Instances

ECS

RDS

ElastiCache

CSVTSVJSON

SQS

IM-DMP


S3

3rd Party Partners

Spot Instances

Spot Fleet

ECS

IM-DMP - Elasticsearch on Spot Instances

Elasticsearch: Approx. 400 million IDsv Extract IDs with

v Keyword (by browsing history)v Segmentv User Agentv IP addressv Geo

https://www.slideshare.net/im_docs/elasticsearch-48873206


Elasticsearch on Spot Instancesv approx. 500vCPUs for Analytics workload

Over 8vCPUs i3 Instances


Intimate MergerSmall engineering team delivers Big results!

Masafumi Osogoe & Kazuki Matsuda

Kazuki Masafumi

Bulbit - Adways company

http://adways.kr/https://www.adways.net

Bulbit - Party Track & UNICORN

68

UNICORN - Full Automated Marketing Platform

Greatly Skilled Engineers from China JvHailin HuvXiaoyi Qu


ü How do you feel about working on AdTech in Japan?

ü What are you focusing on?

HailinXiaoyi

Greatly Skilled Engineers from China JvHailin HuvXiaoyi Qu


Itʼs like a “Gold Mine”ü Day-by-Day EvolutionüAchieving Goals with latest

Big Data Technologiesü Utilize “Amazon Athena”

in a massive way! HailinXiaoyi

UNICORN - Architecture

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Architecture

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Real-Time Bidding

From Ruby to Golang“Speed is King”

in Real-Time Bidding

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Data Analysis

v Extract data for Machine Learning every 30minv Ad-Hoc Big Data Analysis

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Machine Learning

v w/ Minimum Librariesü No Heavy Frameworkü As Fast As Possible!!

v Making Steady Effortü Plan-Do-Check-Actü Parameter Tuningü A/B Testing

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Machine Learning

For Real-Time Bidding, Bidding servers load “Trained Models” into Memory

Auto Scaling

Up to 200instances

Athena

Redshift

Deep Learningon EC2

S3

UNICORN - Big Data Technology

v Right Technology in the Right Placev Recently in favor with “Apache Flink”

AdTech use-cases in Japan

Let's meet up at re:Invent 2017!!

https://reinvent.awsevents.com/

Let's meet up at re:Invent 2017!!

https://reinvent.awsevents.com/

@Kimchi Korean BBQhttp://www.kimchilasvegas.com/

감사합니다!!