transforming one small step at a time: optimisation & testing
DESCRIPTION
Digital transformation isn't a project - it's continuous: small experiments which lead to a culture of improvement through learning. In the presentation, Principle Consultant Jonathan Whiteside and UX & Site Optimisation Consultant Kyle Cassidy discuss how to plan and test for continual user experience improvements, to ensure your optimisation goals are aligned with your organisation's overall annual objectives. They also share practical measuring and reporting tips. First presented at the Figaro Digital Conference on 27 November 2014.TRANSCRIPT
Building Blocks
Transforming One
Small Step At A Time
Building Blocks www.building-blocks.com
Building Blocks
A talk about testing?!?
Jonathan WhitesidePrincipal Consultant
Kyle CassidyUX & Optimisation Consultant
Building Blocks
It’s much cheaper to servethan acquire
Companies that embrace optimisation to make decisions increase conversion by 100%
Source: Adobe
Building Blocks
A talk about testing?!?“Insanity: doing the
same thing over and over again and expecting
different results”- Albert Einstein
Building Blocks
Supports improvement
Organisational Digital Maturity
Confidence
Business case justification
De-risk projects
Maximise campaign effectiveness
Incremental conversion increases
Support business-wide commercial decisions
Building Blocks
What we’re going to talk about
1. What?
2. Why?
3. Examples
4. How?
Building Blocks
So, what is site optimisation?
Building Blocks
Optimisation includes:
• A/B & MVT testing• Analytics• CRO• User journeys• Usability• Information architecture• Traffic sources• Segmentation• User experience design• Personalisation
Building Blocks
Optimisation is for everyone
Marketing Commercial
Web Operations/
IT
Optimisation
Building Blocks
It’s much cheaper to servethan acquire
It’s much cheaper to serve, than acquire…
Building Blocks
Clicks
PPC Cost per click
Total PPC spend
Conversion rate
Customers won
Cost per acquisition
Average profit per customer
Total profit generated
ROI
Simple numbers…
Non Optimised website Optimised website
100
£0.06
£66
1%
1
£66
£40
£40
-40%
100
£0.06
£66
5%
5
£13
£40
£200
200%
Building Blocks
Experiments
Building Blocks
A/B MVT
Examples
Building Blocks
• 1 element on the page
• Multiple variants
• Tested against a ‘control’
• Evenly distributed traffic
• Measured over time
What is A/B Testing?
Building Blocks
Examples of things to test
• Buttons• Form placement• Images• Copy changes• Icons• Video• Navigation• Banners
Building Blocks
Building Blocks
A/B testing: upgrades page
Conversion
20.47%
Building Blocks
Testing variantsL
eve
l 2
-g
rey
ba
ckg
rou
nd
a
nd
up
gra
de
pro
mis
e
Lev
el
3 –
wh
ite
ba
ckg
rou
nd
a
nd
up
gra
de
pro
mis
e
Lev
el
4 –
gre
y b
ackgro
und
Lev
el
5 –
white b
ackgro
und
Co
ntr
ol
Building Blocks
The results
ExperimentSubmitted form Conversion Rate
Lift Chance to beat control
Control20.47%
+/- 0.84%
Level 234.31%
+/- 0.98%+67.59%
100% chance to beat control
Level 334.94%
+/- 0.98%+70.64%
100% chance to beat control
Level 434.89%
+/- 1.00%+70.40%
100% chance to beat control
Level 534.21%
+/- 0.98%+67.07%
100% chance to beat control
Conversion
34.94%
Building Blocks
20.47% 34.94% 70.69%
Additional RecordsCaptured each month
Controlconversion rate
Winnerconversion rate
Conversion rate uplift
17,800
The impact of the conversion uplift
(£20 per lead) Per month in upgrade leads£356,000
Building Blocks
• Multiple elements
• Multiple variants
• Tested at the same time
• Tested against a ‘control’
• Traffic is distributed evenly
• Measured over time
What is MVT Testing? Opt. A
Opt. B
Opt. C
Opt. D
Building Blocks
Co
ntr
ol
Lev
el3
Lev
el 2
Lev
el 4
Testing variants
Building Blocks
ExperimentSubmitted form Conversion Rate
Lift Chance to beat control
Desktops
Control27.55%
+/- 0.69%-
Level 229.931%
+/- 0.71%+8.63%
High significance
Level 329.99%
+/- 0.71%+8.85%
High significance
Level 430.93%
+/- 0.72%+12.25%
High significance
iPads
Control 22.91% +/- 1.43%
-
Level 2 24.80%+/- 1.46%
+8.58%High significance
Level 3 25.63%+/- 1.48%
+12.36%High significance
Level 4 26.73%+/- 1.52%
+17.36%High significance
Desktops
30.93%iPads
26.73%
Results
Building Blocks
Controlconversion rate
Winnerconversion rate
Conversion rate uplift
The impact of the conversion uplift
27.55%22.91%
30.93%26.73%
+12.25%+17.36%
4,920
Additional RecordsCaptured each month
(£20 per lead) Per month in upgrade leads£98,000
Building Blocks
A/B vs. MVT
A/B MVTBest for Large scale re-designs Incremental tests
Complexity Low High
Variants Known (usually 2-4) Multiple (+25)
Served 1 at a time Random order
Output Result of the test Result of the elements
Time to execute Low High
Time to run (traffic/variant dependent)
Shorter Longer
Building Blocks
Not always successful…
Building Blocks
That all sounds great, where do I start?
Building Blocks
A talk about testing?!?
Common excuses to ignore optimisation
52%
36%
28%
26%
25%
23%
22%
21%
10%
9%
3%
0% 15% 30% 45% 60%
Lack of resources
Lack of budget
Conflict of interest between departments
Lack of strategy
Poor technology
Poor intergration between systems
Siloed Organisation
Company culture
Lack of ownership
Reliance on third parties
Other
Source: eConsultancy
Building Blocks
It’s much cheaper to servethan acquire90:10
Building Blocks
90% 10% PlanningProcessesPeoplePresentation
Technology
Time investment
Building Blocks
http://chiefmartec.com/
There are lots of tools…
Building Blocks
Objectives
Campaignsupport
Mitigating risk
Developing a testing strategy
Justifying business cases
Improving site usability
Building Blocks
Test Ideation & Prioritisation
Writing Test Plan and
performing QA
Test Live -Monitoring
Results
3 parallel tracks
Building Blocks
What skills do I need?
10%
40% 40%
Presentation skills Technical
10%
Analytically minded
UX/Creative/Copywriting
Building Blocks
How to secure buy-in
• Make sure data is clean, consistent and trusted.
• Use data to identify where to prioritise your optimisation roadmap.
• Define your main optimisation conversion event and how you will use it to calculate ROI.
• Ensure that it aligns to your organisation’s KPIs.
• Invite stakeholder sign-off to communicate that testing will help them reach their goals.
Building Blocks
It’s much cheaper to servethan acquire
Companies with significant increases in sales are completing 6.45 A/B & MVT tests per month
Source: eConsultancy
Building Blocks
• Valuable at every stage
• Test small, test often
• Build business/change cases based on known value
• Lean, agile with little waste
• Try new things, experiment and measure
• Use results as leverage to be bold
Transforming with optimisation
Building Blocks
Building Blocks
Thank you!
building-blocks.com