Евгений Цымбалов, webgames - Методы машинного обучения для...

Game analytics: ML to the rescue!

Evgenii Tsymbalov, Data Scientist

Who we are

WebGames(“WG”) is one of Russia’s largest developers and publishers of free-to-play gamesPlatforms: FB, iOS/ Android, game\social platforms (VK, OK, MW, Congregate, Steam)Daily audience of over 400K playersData: ~80M records per day

Game analytics

Marketing analytics

In-game analytics

Churn predictionRetargetingRevenue predictionUser classification

A/B testingBalanceRecommendationsUser/content classification

Churn and retargeting

Churn

Retargeting

Find users who are about to stop playingGive them bonuses…PROFIT!

Channels: app-to-user notifications, messages, mail

Find and support users for retargetingChannels: traffic control

Revenue prediction

Costumer LTV (Lifetime Value) - estimate of overall profit from the entire future relationship with a customer. Applications:

indicator of project healthiness; advertising actions planning; in-game events planning.

It is important to estimate LTV not in general (platform of project), but for different cohorts or even every individual player.

LTV: methodology and assumptions

Estimating LTV-100 (may vary for project). User’s actions determined by his or her behavior in first

30 days after registration => 30 different models, kth for users who plays k days; Tracking only last year data

General multistage model:Classification (going to pay or not)Regression (revenue prediction)Additional low-level classifiers, such as events, holidays, etc.

LTV: accuracy metrics

TA (total accuracy) = . TA = 1 for perfect predictor.

RAE-d (relative absolute error on day d) =. This equals to zero for

perfect predictor.

Here, - LTV-100 forecast for i-th player, – total revenue on day d for i-th player. TA is a main indicator for marketing department, while RAE is widely used to compare models’ performance on different days.

LTV models: kNN + cohorts

User classification

User classification

A/B testing

Classic approach: fixed group size, results after full filling.

Bayesian approach: prior distribution changes over time with test results using Bayes theorem.

Bayesian A/B Testing at VWO, Chris Stucchio, 2015

Balance and recommendations

Classic approach: game-designers with Google Spreadsheets.Better approach: modeling.

Rule-based approachMidgame support based on classificationContent recommendations.

Static case

Dynamic case

Balance: rule-based approach

Balance: midgame support

Content clustering

Instead of conclusion: what helps us

Instead of conclusion: what helps us

Questions?

evgenii.tsymbalov@corpwebgames.com

Вопрос из зала: конкретные цифры

Вопрос из зала: какие алгоритмы?