webshop wednesday #11: best practices and lessons learned from 100,000 a/b tests, martijn...

Best Practices and Lessons Learned from

100,000 A/B TestsA/B TestsMartijn Geerlings

100,000

Thursday, November 28, 13

Is de (klant) reis...

belangrijker dan de bestemming?

Thursday, November 28, 13

2009! 2010! 2012!2011! 2013!

Thursday, November 28, 13

Lessons Learned

Thursday, November 28, 13

Lesson #1:

Define success .

Thursday, November 28, 13

Thursday, November 28, 13

Lesson #1:

Define success .

Thursday, November 28, 13

Lesson #1:

Define quantifiable success metrics.

Thursday, November 28, 13

Thursday, November 28, 13

Visitvisitors per day

Thursday, November 28, 13

Visitvisitors per day

Emailsign up

rate

Thursday, November 28, 13

Visitvisitors per day

Emailsign up

rate$

$ per recipient

Thursday, November 28, 13

Visit Email $sign up

rate $ per recipientvisitors per day

Thursday, November 28, 13

Media

Button

Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Splash page experimentVariations:

Thursday, November 28, 13

Button: “Sign Up”Thursday, November 28, 13

Button: “Learn More”Thursday, November 28, 13

Button: “Join Us Now”Thursday, November 28, 13

Button: “Sign Up Now”Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment

Thursday, November 28, 13

Media: “Get Involved”Thursday, November 28, 13

Media: “Family”Thursday, November 28, 13

Media: “Change”Thursday, November 28, 13

Media: “Barack’s Video”Thursday, November 28, 13

Media: “Springfield Video”Thursday, November 28, 13

Media: “Sam’s Video”Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment

Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment

Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment

Thursday, November 28, 13

Splash page experiment

Thursday, November 28, 13

Splash page experiment

Thursday, November 28, 13

Splash page experiment

Thursday, November 28, 13

Email Subscriptions Volunteers Amount Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Thursday, November 28, 13

Email Subscriptions Volunteers Amount Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Thursday, November 28, 13

Email Subscriptions Volunteers Amount Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment

Thursday, November 28, 13

Lesson #1:

Define quantifiable success metrics.

Thursday, November 28, 13

Lesson #2:

Less is more. Reduce choices.

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

+8% engagement

Thursday, November 28, 13

+43.8% Purchases

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

+14.8% conversion rate

Thursday, November 28, 13

Lesson #2:

Less is more. Reduce choices.

Thursday, November 28, 13

Lesson #3:

Words matter. Focus on your call to action.

Thursday, November 28, 13

Donation button experiment

Variations

Thursday, November 28, 13

Donation button experiment

VariationsNever

Signed UpSigned Up,

Never DonatedPreviously Donated

0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

+14.6% clicks

Thursday, November 28, 13

Lesson #3:

Words matter. Focus on your call to action.

Thursday, November 28, 13

Lesson #4:

Fail fast.

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

-10% conversions

Thursday, November 28, 13

vs.

Thursday, November 28, 13

vs.

-10% conversions

Thursday, November 28, 13

Lesson #4:

Fail fast.

Thursday, November 28, 13

Lesson #5:

Seek the Global Maximum.

Thursday, November 28, 13

Thursday, November 28, 13

Thursday, November 28, 13

Thursday, November 28, 13

Thursday, November 28, 13

Thursday, November 28, 13

+0.5% Retention +1.4% Engagement

Thursday, November 28, 13

Lesson #5:

Seek the Global Maximum.

Thursday, November 28, 13

1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Seek the Global Maximum.

Lessons Learned

Thursday, November 28, 13

Thursday, November 28, 13

One more thing...

Thursday, November 28, 13

6000+ happy testers

Start Today!

Thursday, November 28, 13