numbers and limits: balancing data and design in product management
TRANSCRIPT
Numbers and LimitsBalancing data and design
April 17, 2016
Who is this guy?
Overview
● What’s the problem?
● How do we fix it?
● Getting better at both
Data mindset vs Design mindset
A data mindset is... A design mindset is...
● focused on users as groups ● focused on users as individuals
● measurable ● intuitive
● incrementally focused ● blue-sky thinking
● focused on identifying and weighing different solutions
● focused on identifying, then solving, the right problem
Common problems
“Numbers don’t tell the whole story. How can you boil down someone’s experience into a single metric?”
“The a/b test results were obvious; why can’t we all just move forward with the winning variant?”
“I don’t know whether to trust my head or my heart.”
When data thinking and design thinking disagree, what do you do?
Common problems
What is a PM?
Analytical
User empathetic Organized
What does a PM do?
Analysis
Product design
Project management
Conflict boundaries
Analysis
Product design
Project management
This is where design lives...
So conflicts arise here...
and here
Overview
● What’s the problem?
● How do we fix it?
● Getting better at both
General principles
● right situation, right mindset
● settling differences of opinion
● data skepticism
Right situation, right mindset
● What might tip you toward relying on data, and what might tip you toward design thinking?○ userbase○ product maturity○ internal stakeholders
Settling differences of opinion
● what objective information can you gather before building anything?
● how do you design features and roadmaps to generate answers for you?
Data skepticism
Control (original) Variant
New feature XYZ is here! Do you want to try it out now?
Try it Cancel
New feature XYZ is here! Do you want to try it out now?
Try itCancel
This is an example; no actual users were harmed in the making of this slide
Data skepticism
● CTR increased from 15% to >19%
● this is huge! we’re gonna be rich!
● where else can we apply this learning?
Also an example
Data skepticism
Also an example
Release process: in theory
Write specs In dev
Write specs
Live!
In dev
January February March
Release 1.0
Release 2.0
April
Live!
Write specs In devRelease 3.0
May
Live!
In practice...
Write specs In dev
Write specs
Live!
In dev
January February March
Release 1.0
Release 2.0
April
Write specs In devRelease 3.0
May
Live!
Write specsIn dev
X X X
1. Design and specwriting takes longer than planned, but team isn’t happy and aligned with the output...
2. ...causing dev to start late and take longer because team is thrashing on release’s design while engineers are working
3. Meanwhile, your next release hasn’t started because 2.0 is dragging
One solution
(If needed) Explore design
Add extra time for exploring and aligning on problem and solution space; get stakeholders to agree upfront
Write specs In dev
Write specs
Live!
In dev
January February March
Release 1.0
Release 2.0
April
Live!
Write specs In devRelease 3.0
May
Live!
Overview
● What’s the problem?
● How do we fix it?
● Getting better at both
Hybrids are great
Further reading
● Andrew Chen’s blog, http://andrewchen.co
● Good PM, Bad PM by Ben Horowitz
● Don’t Make Me Think
● The Design of Everyday Things