badoo: cross platform mobile test automation and continuos delivery
TRANSCRIPT
Badoo: Cross pla,orm mobile test automa3on and con3nuous delivery
Sathish Gogineni Mobile QA Automa4on Lead
Raise your hands if…. • You’re working in Mobile pla=orm?
• You have apps on mul4ple mobile pla=orms?
• You’re aware of the test automa4on tools in Mobile?
• You are using func4onal test automa4on?
• You’re using “Cross pla=orm” test automa4on?
Who is Badoo?
• A social networking service
• 241 million users
• Available in 190 countries.
• Support 45 languages on mobile
• 250 employees based in London and Moscow
Mobile Test Automa3on at Badoo • A single test automa4on code base
– Run across two products – Run on three pla=orms – Run on phones and tablets
• We have – 68% func4onal coverage for Badoo in Android & iOS ( 390 scenarios )
– 76% func4onal coverage for HotOrNot in Android & iOS ( 250 scenarios )
– 52% func4onal coverage in MobileWeb ( 165 scenarios ) – And in Windows pla=orm we have 0%
Mobile cross pla,orm test automa3on?
• Write once , Run anywhere(?)
• Write test specifica4ons
• Implement the tests at high level
• Implement the pla=orm/applica4on UI code
Cross pla,orm stack in BDD
Scenarios
Feature
Step defini3ons
Android Page iOS Page
Test Specifica4on
High level implementa4on Pla=orm specific
Why cross pla,orm test automa3on?
• Business logic is same. It is only interac4on with UI is change
• Pooled resources
• More reusable code. Less maintenance
What should be considered?
• Who is wri4ng the test cases? App developers / QA developers?
• Which mobile pla=orms?
• Wri4ng in language other than pla=orm specific
Mobile Cross Pla,orm Tools
• Instrument based tools
• Image Recogni4on tools
– Calabash – Appium – Monkey Talk
– eggPlant – seetest
What is Calabash?
• A framework wriben in “Ruby”
• Has client-‐server architecture using JSON
• Integrated with cucumber
• Supports Android and iOS pla=orms
Calabash : Pros & Cons • Pros – Has well structured client framework – Supports backdoor opera4ons to interact with Applica4on code
• Cons – Can’t run on “Produc4on” build in iOS – Have to use Ruby programming language – Forced to write test cases in BDD ( ‘Given When Then’ )
What is Appium? • Is a Server wriben in Node.js using JavaScript
• Uses pla=orm-‐provided frameworks under the hood
• Supports Web driver JSONWire protocol
• Supports iOS, Android, Mobile Web and Hybrid applica4ons
Appium : Pros & Cons • Pros – Tests can be wriben in any ( most of ) programming languages
– Good community support – Can use same produc4on builds for tes4ng
• Cons – Lible bit “buggy”
Con3nuous Delivery is…
A prac4ce of crea4ng, tes3ng and releasing stable applica4on frequently in smaller chunks
Con3nuous Test Automa3on
Development Branch
Feature Tests
Smoke Tests
Nightly Builds
Nightly Builds
Implement Feature tests
Con3nuous Test Automa3on
Development Branch
Integra4on Branch
Integra4on Tests
Master
Feature 1
Feature 2
Con3nuous Test Automa3on Development Branch
Integra4on 2
Integra4on 1
Master Branch
Nightly Builds
Nightly Builds
Con3nuous Test Automa3on Development Branch
Integra4on 2
Release Tests
Integra4on 1
Master Branch
Release Branch
Con3nuous Test Automa3on Development Branch
Integra4on 2
Integra4on 1
Master Branch
Release Branch
Release Tests
Release Tests
How we started?
• A single build machine with Jenkins
• Running on Android emulator and iOS simulator
• Run the tests only on “master” branch every night
What we have today?
• Distributed CI server with 12 mac-‐mini agents
• Each Agent has Android and iOS device connected
• Running the tests on Real devices
• Integrated with JIRA
What we will do in future?
• More devices to each agent and Parallelizing test suite
• Using VMs for scaling build agents
Summary
• Worth wri4ng cross pla=orms tests. It Ensure behavior is same in every pla=orm
• Mobile Test Automa4on tools are s4ll evolving
• And, they have limita4ons
Nevertheless… • We reduced Regression tes4ng to a day for every release ( 66% 4me save )
• 3X more releases to the app stores
• More stable applica4on. Crash rate reduced from 6% to 2%