mobile @ facebook - qconsf 2013

Mobile @ Facebook

Christian Legnitto (christianl@fb.com)Mobile release engineering

Who is this guy?

Who is this guy?

Facebook• Manage Mobile Release Engineering

• I’m hiring

• Run the iOS and Android releases

• Ran Facebook.com web pushes for ~3 months

Who is this guy?

Facebook

Mozilla

• Manage Mobile Release Engineering

• I’m hiring

• Run the iOS and Android releases

• Ran Facebook.com web pushes for ~3 months

• Firefox release manager

• Helped design & implement 6 week releases

• Desktop ( 3 platforms ) & mobile ( Android )

• Still a community member

Who is this guy?

Facebook

• Jaguar → Lion updates

• OS X pre-release updates

• Safari, Xcode, Java, iApps

• Build system, scrubber, PM tools

Mozilla

Apple

• Manage Mobile Release Engineering

• I’m hiring

• Run the iOS and Android releases

• Ran Facebook.com web pushes for ~3 months

• Firefox release manager

• Helped design & implement 6 week releases

• Desktop ( 3 platforms ) & mobile ( Android )

• Still a community member

874,000,000 MAUs

507,000,000 DAUs

native & web

supports iOS6+

iPads & iPhones

+1M beta testers

+20K alpha

testers

native & web

supports Froyo+

15 apks

How we USED to develop

Mobile core team owned mobile

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

. . . . . .

Messages team

Events team

Photos team

Android core team

iOS core team

Doesn’t SCALE

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

20

10

30

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

20

10

30

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

(numbers made up)

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

20

10

30

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

20

10

30

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

2010

30

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

201030

20

20

Messages team

Events team

Photos team

Android core team

iOS core team

60

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

201030 20

20

Messages team

Events team

Photos team

Android core team

iOS core team

60

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

201030 20 20

Messages team

Events team

Photos team

Android core team

iOS core team

60

Doesn’t produce the BEST product

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

Messages team

Events team

Photos team

Android core team

iOS core team

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

Messages team

Events team

Photos team

Android core team

iOS core team

product experts

Mobile core team owned mobile

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

Messages team

Events team

Photos team

Android core team

iOS core team

product experts platform experts

How we USED to ship

Feature-based releases

Feature-based releases

Decide on features

Feature-based releases

Decide on features

Furiously work on them

Feature-based releases

Decide on features

Furiously work on them

Test

Feature-based releases

Decide on features

Furiously work on them

Test

Ship when all features are done

All features are PRIORITY #1

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Decide on features

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Decide on features

1. Emoji 2. Green dot 3. Naming convos

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

1. Automatic upload 2. Tag suggestions 3. Change cover photo

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

1. Automatic upload 2. Tag suggestions 3. Change cover photo

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

1. Automatic upload 2. Tag suggestions 3. Change cover photo

1. Emoji1. Reply inline1. Automatic upload

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

1. Automatic upload 2. Tag suggestions 3. Change cover photo

1. Emoji1. Reply inline1. Automatic upload

2. Green dot2. Birthday reminder2. Tag suggestions

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji 2. Green dot 3. Naming convos

1. Reply inline 2. Birthday reminder 3. Gifts integration

1. Automatic upload 2. Tag suggestions 3. Change cover photo

1. Emoji1. Reply inline1. Automatic upload

2. Green dot2. Birthday reminder2. Tag suggestions

3. Naming convos 3. Gifts integration3. Change cover photo

The market and company CHANGES RAPIDLY

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

. . .

Decide on features

1. Emoji2. Reply inline3. Automatic upload

4. Green dot5. Birthday reminder6. Tag suggestions

7. Naming convos 8. Gifts integration9. Change cover photo

Don’t know when code SHIPS

Can’t judge risk vs rewardFeature-based releases

Can’t judge risk vs rewardFeature-based releases

80% done?

Can’t judge risk vs rewardFeature-based releases

Good enough80% done?

Confuses DEV and RELEASE

Development processCreate the best product possible

→ Focus on improving

Development processCreate the best product possible

→ Focus on improving

Release processShip finished work in a timely manner

→ Focus on not regressing

Took lessons learned from DESKTOP WEB and applied

them to MOBILE

How we develop NOW

Old and busted

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Android

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS

. . . . . .

Messages team

Events team

Photos team

Android core team

iOS core team

New hotness

. . .

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Desktop Web Android iOS

. . . . . .

Messages team

Events team

Photos team

Android core team

iOS core team

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

Photo picker

Photo albums

Chat

Group messages

Upcoming events

Birthdays

iOS and AndroidEngineers committing mobile code

How we ship NOW

featuresqualityschedule

pick 2featuresqualityschedule

pick 2featuresqualityschedule

Rule #1Date-based releases

We ship on time

Rule #2Date-based releases

Make usersno worse off

Rule #3Date-based releases

There’s alwaysthe next one

Rule #4Date-based releases

Retreat to safety

4 weeks of development

master

3.5 weeks of stabilization

4 weeks of development

native mobile

release branch

Automated cut every 4 weeks

RC

1 week of development

trunk

Stabilize

1 week of development

desktop web

Weekly Twice Daily

release branch

Automated cut every 1 week

Release branch

4 weeks of development

master

3.5 weeks of stabilization

4 weeks of development

native mobile

release branch

Automated cut every 4 weeks

RC

master

native mobile

release branch

Dogfooding

How we write CODE

GitSource control

>300 engineers

>150,000 files

>100,000 commits

PhabricatorCode review

BuildbotContinuous integration

Facebooki386Builds Facebook

armMessenger

armMessenger

i386 ...

Lint / static analysis regex clang android findbugs ...

Tests OCUnit JUnitsnapshot robolectric ...

https://github.com/facebook/ios-snapshot-test-case

BuildbotContinuous integration

Before diff(optional)

Create diff(mandatory)

Diff updated(mandatory)

Landing queue(optional)

Git commit(mandatory)

Mac minisContinuous integration

Android

Buck - build toolBuilding & running tests

• Fast

• Scalable and extensible

• “buck clean is a bug”

• Android and generic Java support

• Distributed Cassandra build cache

• buckd for background builds

Buck - build toolBuilding & running tests

• Fast

• Scalable and extensible

• “buck clean is a bug”

• Android and generic Java support

• Distributed Cassandra build cache

• buckd for background builds

Buck - build toolBuilding & running tests

“Gerrit Code Review recently switched its build process from

Maven to Buck, reducing build time by 67-98% while improving build reproducibility, maintainability and developer happiness.

The Gerrit project migrated itself out of the Maven tarpit into a

better build.” - Shawn Pearce (Google)

Buck - join us!Building & running tests

Start using Buck:

http://facebook.github.io/buck/

Get help in the discussion group:

https://groups.google.com/forum/#!forum/buck-build

File bugs and submit patches on GitHub:

https://github.com/facebook/buck

iOS

xctool - build toolBuilding & running tests

Makes iOS & Mac continuous integration easier.

• CLI tool, runs tests just as Xcode.app would

• Can parallelize test runs

• Can emit JSON objects for every build/test event

• Reporters for Phabricator, JUnit XML

• Attractive, concise output

xctool - “pretty” reporterBuilding & running tests

xctool - “json-stream” reporterBuilding & running tests

xctool - how it worksBuilding & running tests

xcodebuild-shim.dylibxcodebuild

xctool

test-shim.dylibotest

test-shim.dylibxctest

(SenTestingKit)

(XCTest)

Injected

Injected

Injected

xctool - join us!Building & running tests

On GitHub:github.com/facebook/xctool

On Homebrew:brew install xctool

Also available on Travis-CI

atosCrash reports

atoslCrash reports

atoslCrash reports

github.com/facebook/atosl

Changed how we DEVELOPChanged how we SHIPChanged how we WRITE CODE

Kept our CULTURE

Learn from our experience and use our toolsFacebook is mobile

Links to mentioned awesomeness

Contact me christianl@fb.com, https://fb.me/legnitto, @LegNeato

Buck http://github.com/facebook/buck

XCTool http://github.com/facebook/xctool

atosl http://github.com/facebook/atosl

Snapshot Test http://github.com/facebook/ios-snapshot-test-case

Mobile @ Facebook

Christian Legnitto (christianl@fb.com)Mobile release engineering