transforming one small step at a time: optimisation & testing

Building Blocks

Transforming One

Small Step At A Time

Building Blocks www.building-blocks.com

Building Blocks

A talk about testing?!?

Jonathan WhitesidePrincipal Consultant

Kyle CassidyUX & Optimisation Consultant

Building Blocks

It’s much cheaper to servethan acquire

Companies that embrace optimisation to make decisions increase conversion by 100%

Source: Adobe

Building Blocks

A talk about testing?!?“Insanity: doing the

same thing over and over again and expecting

different results”- Albert Einstein

Building Blocks

Supports improvement

Organisational Digital Maturity

Confidence

Business case justification

De-risk projects

Maximise campaign effectiveness

Incremental conversion increases

Support business-wide commercial decisions

Building Blocks

What we’re going to talk about

1. What?

2. Why?

3. Examples

4. How?

Building Blocks

So, what is site optimisation?

Building Blocks

Optimisation includes:

• A/B & MVT testing• Analytics• CRO• User journeys• Usability• Information architecture• Traffic sources• Segmentation• User experience design• Personalisation

Building Blocks

Optimisation is for everyone

Marketing Commercial

Web Operations/

IT

Optimisation

Building Blocks

It’s much cheaper to servethan acquire

It’s much cheaper to serve, than acquire…

Building Blocks

Clicks

PPC Cost per click

Total PPC spend

Conversion rate

Customers won

Cost per acquisition

Average profit per customer

Total profit generated

ROI

Simple numbers…

Non Optimised website Optimised website

100

£0.06

£66

1%

1

£66

£40

£40

-40%

100

£0.06

£66

5%

5

£13

£40

£200

200%

Building Blocks

Experiments

Building Blocks

A/B MVT

Examples

Building Blocks

• 1 element on the page

• Multiple variants

• Tested against a ‘control’

• Evenly distributed traffic

• Measured over time

What is A/B Testing?

Building Blocks

Examples of things to test

• Buttons• Form placement• Images• Copy changes• Icons• Video• Navigation• Banners

Building Blocks

Building Blocks

A/B testing: upgrades page

Conversion

20.47%

Building Blocks

Testing variantsL

eve

l 2

-g

rey

ba

ckg

rou

nd

a

nd

up

gra

de

pro

mis

e

Lev

el

3 –

wh

ite

ba

ckg

rou

nd

a

nd

up

gra

de

pro

mis

e

Lev

el

4 –

gre

y b

ackgro

und

Lev

el

5 –

white b

ackgro

und

Co

ntr

ol

Building Blocks

The results

ExperimentSubmitted form Conversion Rate

Lift Chance to beat control

Control20.47%

+/- 0.84%

Level 234.31%

+/- 0.98%+67.59%

100% chance to beat control

Level 334.94%

+/- 0.98%+70.64%

100% chance to beat control

Level 434.89%

+/- 1.00%+70.40%

100% chance to beat control

Level 534.21%

+/- 0.98%+67.07%

100% chance to beat control

Conversion

34.94%

Building Blocks

20.47% 34.94% 70.69%

Additional RecordsCaptured each month

Controlconversion rate

Winnerconversion rate

Conversion rate uplift

17,800

The impact of the conversion uplift

(£20 per lead) Per month in upgrade leads£356,000

Building Blocks

• Multiple elements

• Multiple variants

• Tested at the same time

• Tested against a ‘control’

• Traffic is distributed evenly

• Measured over time

What is MVT Testing? Opt. A

Opt. B

Opt. C

Opt. D

Building Blocks

Co

ntr

ol

Lev

el3

Lev

el 2

Lev

el 4

Testing variants

Building Blocks

ExperimentSubmitted form Conversion Rate

Lift Chance to beat control

Desktops

Control27.55%

+/- 0.69%-

Level 229.931%

+/- 0.71%+8.63%

High significance

Level 329.99%

+/- 0.71%+8.85%

High significance

Level 430.93%

+/- 0.72%+12.25%

High significance

iPads

Control 22.91% +/- 1.43%

-

Level 2 24.80%+/- 1.46%

+8.58%High significance

Level 3 25.63%+/- 1.48%

+12.36%High significance

Level 4 26.73%+/- 1.52%

+17.36%High significance

Desktops

30.93%iPads

26.73%

Results

Building Blocks

Controlconversion rate

Winnerconversion rate

Conversion rate uplift

The impact of the conversion uplift

27.55%22.91%

30.93%26.73%

+12.25%+17.36%

4,920

Additional RecordsCaptured each month

(£20 per lead) Per month in upgrade leads£98,000

Building Blocks

A/B vs. MVT

A/B MVTBest for Large scale re-designs Incremental tests

Complexity Low High

Variants Known (usually 2-4) Multiple (+25)

Served 1 at a time Random order

Output Result of the test Result of the elements

Time to execute Low High

Time to run (traffic/variant dependent)

Shorter Longer

Building Blocks

Not always successful…

Building Blocks

That all sounds great, where do I start?

Building Blocks

A talk about testing?!?

Common excuses to ignore optimisation

52%

36%

28%

26%

25%

23%

22%

21%

10%

9%

3%

0% 15% 30% 45% 60%

Lack of resources

Lack of budget

Conflict of interest between departments

Lack of strategy

Poor technology

Poor intergration between systems

Siloed Organisation

Company culture

Lack of ownership

Reliance on third parties

Other

Source: eConsultancy

Building Blocks

It’s much cheaper to servethan acquire90:10

Building Blocks

90% 10% PlanningProcessesPeoplePresentation

Technology

Time investment

Building Blocks

http://chiefmartec.com/

There are lots of tools…

Building Blocks

Objectives

Campaignsupport

Mitigating risk

Developing a testing strategy

Justifying business cases

Improving site usability

Building Blocks

Test Ideation & Prioritisation

Writing Test Plan and

performing QA

Test Live -Monitoring

Results

3 parallel tracks

Building Blocks

What skills do I need?

10%

40% 40%

Presentation skills Technical

10%

Analytically minded

UX/Creative/Copywriting

Building Blocks

How to secure buy-in

• Make sure data is clean, consistent and trusted.

• Use data to identify where to prioritise your optimisation roadmap.

• Define your main optimisation conversion event and how you will use it to calculate ROI.

• Ensure that it aligns to your organisation’s KPIs.

• Invite stakeholder sign-off to communicate that testing will help them reach their goals.

Building Blocks

It’s much cheaper to servethan acquire

Companies with significant increases in sales are completing 6.45 A/B & MVT tests per month

Source: eConsultancy

Building Blocks

• Valuable at every stage

• Test small, test often

• Build business/change cases based on known value

• Lean, agile with little waste

• Try new things, experiment and measure

• Use results as leverage to be bold

Transforming with optimisation

Building Blocks

Building Blocks

Thank you!

building-blocks.com