startup kpis and a/b testing
TRANSCRIPT
Startup KPIs & A/B TestingWhat they are, why you should use them, & how to set them up
[email protected]/in/jeffmcclelland
Hi, I’m Jeff!
Integral Designs Bell Canada ING Bank Skype → Microsoft TransferWise
MSc - Informa-
tion Science
Bachelors - Business
data → data → data → data → data → data → data → data → data → data → data → data → data
Workshop Topics
What are you trying to do?Why focus on KPIs?What are well-defined KPIs?Where does A/B testing fit in?
KPIs + A/B testing support the org’s learning
You win if you learn faster● Smart KPIs are widely
applicable across your business
● A/B testing is a specialized tool: great for a small number of problems
Data-driven organization
Focus on learning
Smart KPIsA/B
Testing
You Win
What are you trying to do?
MVP Learning Cycle
MVP - does not mean “minimum viable piece of crap” ● You need to understand who
your customers are and what they want.
● Can the customers use your product?
● Can you do for the customer what you actually promise?
Metrics → feedback → learning
You need to understand your metrics and set realistic targets!This will help you define success and start iterating. For instance:● New users● NPS● Cancellations● Conversion
Iterate
Usually the one that iterates fastest wins!● Prioritising is they key here● Make sure you know what
you effect and measure it!● A/B testing
Why focus on KPIs?
You have a hard job
● You have a limited opportunity to become successful
● Your success depends on how quickly you learn.
● Specifically, how quickly you learn how to deliver: ○ A product customers value○ A sustainable business at scale
You’ll learn faster with good KPIs
Using well-defined KPI’s help you learn faster: ● Which products/features
customers value● How effectively you’re
scaling:○ marketing channels ○ costs○ operations
● Which activities you should focus on
You are here
You want
to get here
Why you should focus on KPIs
● You have a mountain to climb to achieve your vision - KPIs are your guideposts
● Validation: It’s proof that you’re moving forward
● Investors/valuations
What are well-defined KPIs?
The best metric/KPI
We did another thing!
time
the bestmetric
We did a thing!
Metric is stable and flat when you’re not influencing it
5 principles of well-defined KPIs
Directional & Not Ambiguous
IntuitiveImportant Reflects reality
In control of the group it’s measured against
if performance is good, it's good for the user and for the company
Can be explained why it's important for the user and to the company
Proportional to reality, dynamic enough to raise attention (not flat day-after-day), but not so jumpy that you can’t discern signal from the noise
higher or lower is always better all other things being equal
so that the team can commit to and be accountable for improvement (for targets in particular)
Early on in your startup
Do Not Focus OnVanity metrics i.e. total registered users, total volume transferred
Getting statistical significance, split testing (A/B testing)More than a handful of metrics
Focus onThe 3 or 4 most important metrics (KPIs) that indicate your progress. Think about:● Growth & Conversion● How you’re adding value● How much runway you have
Helpful to work backwards from some target state (i.e. 10K new users) and validate how you get there
KPI Examples
Growth - Signups, Downloads, New users, Active Users, Revenue, ProfitEngagement/conversion - feature use, frequency of use, # featuresQuality - Net Promoter Score (NPS), Customer Satisfaction (CSAT)Profitability/Sustainability/cost per user (marketing and/or OPEX)Churn - users stop using your product/service, downgradeVirality/network effects - spreading the word, building the network, market shareSpeed/Efficiency/Backlog/Queues/Time to X - efficiency of your machineRisk/Exposure - Leverage, FX Exposure, Value at Risk, SLA, cash/runway
Task - Define KPIs for your business
Pick a handful of KPIs that drive your business (or area of the company)
Pick a handful of segments
Present it back; give and get feedback
Where does A/B testing fit in?
What are A/B tests?
Everyone believes they know what’s right...but they’re often wrong. ● Everyone has biases● Scientific experiments● A/B Tests can help you learn● If done well they give you
unbiased “proof” of whether A is better than B
When should you use A/B testing?
Work well for optimisation problems - small iterative learning● Product, pricing, messaging,
support all possible
But even if you could, you sometimes shouldn’t as they can be expensive. Learning is expensive.
● Get high volumes and be patient waiting on results
● Don’t corrupt with other overlapping tests
● Know basic statistics
AB testing 101
Are your results significant? tl;dr: no
It’s difficult to achieve statistical significanceYou need to:● Plan/design in advance● Pick the right KPI(s) & scope● Random or pseudo-random
allocation
AB Testing - ballpark testingUseful: http://www.thumbtack.com/labs/abba
Add your variations here
Find out whether it’s significant here
Task - A/B testing
1. Pick up one of each colour at random from the front. These are the parameters of an AB experiment you’ve set up
2. Use those numbers to figure out what numbers to put in each box on here: thumbtack.com/labs/abba/
3. Figure out whether your experiment will be “significant” or not
4. Figure out which variables to change to make it significant (i.e. longer duration?, more visitors?, bigger change?)
Small sample but huge performance increase
A/B Test 1
✔
A/B Test 2
✔
Huge sample but small performance increase
Summary: How to get to significance?
One set is too small
✖
The gap is too small for the number of trials
Summary: How to get to significance?
✖
Summary: How to get to significance?
Success is too rare
✖
The gap is too small for the number of trials
Small sample but huge performance increase
Summary: How to get to significance?✔
Huge sample but small performance increase
✔
One set is too small ✖
✖
Success is too rare ✖
Final notes
RecapData-driven organization
Focus on learning
Smart KPIsA/B
Testing
You Win
● Your goal is to learn quickly● KPIs are important for
learning what to focus on● Well-defined KPIs have five
attributes: important, intuitive, directional, reflect reality, & in your control
● AB testing is powerful when done right, but you have to set it up in a smart way
Learn more
Read
On how to set up KPIs for SaaS companies http://www.forentrepreneurs.com/saas-metrics-2/and any relevant post by Avinash Kaushik
The evolution of data-driven A/B testing & product development at Etsy: http://mcfunley.com/data-driven-products-now
Get in [email protected]: jrmcclelLinkedIn+372 5845 1455
APPENDIX: presenting & using KPIs
Tip: should you use a graph or a table?
GraphsTo focus on one metric and a handful of splits
Trends, change
Missing important context
TablesMany metrics, many splits
Point-to-point comparison & where you need context
Focusing attention
Best use
Challenge
Use when you need
Tip: Colours and Lines
General rule: As little as possible
When you’ve got your data...● Remove all lines/colours and
add logically● Add only when it makes “so
what” more clear
● Every single line, colour, metric and split must have a meaning and add value
● Create an “all other” category● Test: Hide/remove a metric or a
split - see if it changes your interpretation
How?
Useful for any processes that happen over time● sales, engagement,
customer retention, etc.
Help you compare users fairly and isolate behaviour changes from portfolio/mix changes
Tip: Use Cohort metrics when you can
user cohorts time
Tip: be mindful of signal vs noise
Even the same data can tell a completely different storyChoosing what you focus on, and how you look at it matters
Tip: Avoid common KPI pitfalls
If you must denominate, choose the least-bad denominator
Divide by something or target the total?Too broad Too
complex
Unclear link to business goals
Higher & lower values could both be good?
Be careful with ratiosCommon KPI errors
Task - Sketch a one-page dashboard
Using the metrics and slices derived earlier:
Sketch out what you would want to see in a compact (one page or screen) dashboard. Think about:
Frequency (daily, weekly, monthly) & amount of history
Prioritise the most important information
Balance the need for a high-level overview with actionable information
Effective charts/tables
What decisions you will make based on the dashboard (if KPI x does this, I will do that)
Present it back; give and get feedback
APPENDIX: A/B Testing activityPrint out for activity on slide 24
100 visitors / week
1,000 visitors / week
10,000 visitors / week
50,000 visitors / week
1,000,000 visitors / week
1% improvement
3% improvement
5% improvement
7% improvement
15% improvement
1 day
1 week
2 weeks
4 weeks
24 weeks
100,000 visitors / week 10% improvement 4 weeks
1% convert
3% convert
10% convert
15% convert
50% convert
25% convert
10% treated
30% treated
50% treated
90% treated