cloud-based, automated mobile app testing for the enterprise

6/2/15

1

Cloud-‐Based, Automated Mobile App Tes5ng for the Enterprise

Joe Schulz, AVP Emerging Technology

Today’s Speaker

Joe Schulz Orasi So:ware

AVP, Emerging Technology [email protected]

6/2/15

2

Today’s Agenda z  What is Mobile Tes<ng? z  What is Cloud Tes<ng? z  How do I Automate for Mobile?

z  What is Con<nuous Mobile? z  Ques<ons?

What is Mobile Tes5ng?

6/2/15

3

Mobile is BIG

Source: Forbes Magazine, 3/23/2014

7 Billion People in the World

6 Billion have a Mobile Phone

Only 4.5 Billion Have What…?

6/2/15

4

“Only 16% of people will try out an app more than twice…” Digital Trends Magazine, March 2013

“80-‐90% of all apps are used once and then deleted…” Tech Crunch Magazine, March 2013

Source: Dimensional Research, 2014

The New Benchmark: 48% expect a mobile app to respond in two seconds or less

6/2/15

5

Maintaining con<nuity is a huge challenge Layout issue

Device incompaJbility

OS issue Network issue AuthenJcaJon issue

Impact of the network on test results

Capacity > 500+ users

Capacity < 200 users

By not considering actual network condi<ons, testers can get trapped

Without real network condi<ons Real network condi<ons

VS

•  Performance tests under ideal lab condiJons can result in a false sense of security that an applicaJon will perform well aQer deployment

•  By accurately emulaJng real-‐world network condiJons, organizaJons can improve the load Jme of web assets to mobile devices by up to 80% while driving remediaJon costs down by 30%

6/2/15

6

Why did this happen?

Is it AT&T’s fault?

Is it Apple’s fault?

Is it YOUR fault?

The Cost of Not Tes<ng

Source: New York Times Online, February 2012

“…customers using iPads…actually paid twice”

“…problem began in July but went undetected…unRl December”

“Other mobile apps…immune from the problem”

6/2/15

7

What is Cloud Tes5ng?

Introducing the Orasi Mobile TesRng Maturity Model

Is this YOUR mobile lab?

6/2/15

8

Orasi Mobile Tes<ng Maturity

1: Handheld Manual tesJng on borrowed or purchased mobile devices

5 Stages of Mobile Tes<ng Maturity

1: Handheld �  Easy �  Expensive

�  No AutomaJon

2: Crowdsourcing

Manual tesJng by consumers in return for a small reward

6/2/15

9



�  No AutomaJon

2: Crowdsourcing

�  Global �  Unreliable �  No AutomaJon

3: Emulators SoQware version of mobile OS running on a workstaJon



�  No AutomaJon

2: Crowdsourcing


3: Emulators �  Cheap �  AutomaJon �  Unreliable �  Market Delays

4: Tethered

ConnecJng physical devices to a workstaJon with USB or WiFi

6/2/15

10



�  No AutomaJon

2: Crowdsourcing



4: Tethered

�  Real Devices �  AutomaJon

�  Local �  Expensive

5: Cloud ConnecJng to physical devices hosted by a cloud service on a subscripJon basis



�  No AutomaJon

2: Crowdsourcing



4: Tethered

�  Real Devices �  AutomaJon

�  Local �  Expensive

5: Cloud �  Real Devices & Networks �  Emulators �  Scalable �  Global Secure �  AgnosJc AutomaJon �  Cost-‐effecJve �  Up-‐Front Cost

6/2/15

11

Device Cradle Device Cradle

Perfecto Mobile

ScriptOnce AutomaRon for Device-‐AgnosRc ScripRng

UFT Mobile plug-‐in for naRve HP ALM integraRon

Secure Web Interface

Internet or WAN

Device Cradle MobileCloud Service

Mobile Labs

6/2/15

12

How do I Automate for Mobile?

Thousands of combina<ons

32 versions of Android released since 2007 450

acJve models of Android mobile phones

21 versions of iOS released since 2007 7

Common operaJng plaborms

5 Major carrier

Networks in the US

14 primary cell phone

manufacturers worldwide

4 wildly popular

browser technologies

12 common mobile scripJng and programming and languages

6/2/15

13

How Big Is The Challenge? Manual Test Cases to Execute 90 iOS Devices to Test (iPhone 5/5S/5C/6/6S iPad 2/3/4/Mini) 9 iOS Versions to Test (iOS 6/7/8) 3 Test Case ExecuJons 2,430 Average Minutes per Test Case ExecuJon 8

40.5 days 4

162 days = 8 FTE

146 44 FTE

52 FTE

Total iOS Manual Test ExecuJon Time Per Build Testable Releases per sprint Total iOS Manual Test Execu<on Time Per Sprint

Android Device VariaJons to Test Total Android Manual Test Execu<on Time Per Sprint Total Manual Execu<on Time for iOS and Android =

x

x

x

Automa<on Objects

Objects are the building blocks of automaJon

Visual (OCR) vs. NaJve object recogniJon

AutomaJon invokes methods and agributes

6/2/15

14

Hybrid Mobile Automa<on §  Objects in Automa<on

w  NaJve objects preferred but limiJng w  Visual (OCR) objects more flexible but

problemaJc

§  Best alterna<ve is hybrid approach mixing na<ve and visual objects to cover all scenarios w  Provides direct access to applicaJon objects w  Allows interacJon with the device outside

of the custom app

UFT Mobile Automa<on

Full support for advanced UFT funcJonality like run-‐Jme parameters

and data tables

Device-‐agnosJc scripJng allows test reuse across devices

Integrated UFT reporJng provides run results with recordings and

checkpoint screen shots

6/2/15

15

What is Con5nuous Mobile?

Con<nuous Integra<on Frequently merging all developer changes into a shared mainline to avoid merge conflicts

Con<nuous Tes<ng Applying test automaRon to the conRnuous integraRon process

Con<nuous Delivery Extending the automaRon paradigm across the enRre so:ware delivery chain

6/2/15

16

CI/CT/CD for Mobile

Code IDE

Unit Test

Build Tool

Developer Machine

Test Repo

Code Repo

Defect Repo

Development Server

Unit Test

Build Tool

Deploy App

CI Server

Build Server

Mobile Server

Test Tool

Test Server

CI/CT/CD for Mobile

Code IDE

Unit Test

Build Tool

Developer Machine

Test Repo

Code Repo

Defect Repo

Development Server

Unit Test

Build Tool

Deploy App

CI Server

Build Server

Mobile Server

Test Tool

Test Server

Eclipse w/ADT

JUnit Ant

HP ALM

SVN

Jira

JUnit Ant

Cloud Server

Jenkins

Cloud Server

HP UFT

6/2/15

17

C2

Automation never lags by more than one sprint

The “s+1” rule of test automation

Sprint 4 1 Sprint 3 Sprint 2 Sprint 1 Sprint 5

C1

C Component (non-GUI)

G GUI element

B Business process

Stable

C1

G1

C1

G1

G2

C1

G1

C2

G2

C3

B1

C1

G1

C2

G2

C3

B1

G1

C2

Other Considera5ons

6/2/15

18

Public, Private or Mixed? Public Cloud Private Cloud §  Devices available first-‐come, first-‐served

§  Select specific handset as needed §  ReservaRons can be made in advance

§  Ac<vity cleansing important since devices are shared

§  IniRal state not guaranteed

§  Purchase blocks of minutes

§  Device “cradles” reserved for secure, exclusive use

§  Specify any device to be placed in cradle §  Can “swap” devices on request

§  Automa<on requires private cloud §  Ac<vity cleansing less important

§  IniRal state consistent with previous session

§  Purchase cradles by the month

How Many Do I Need? §  “Must” Devices

§  2-‐5 devices used for essenJal and regression tesJng §  AutomaJon is criJcal

§  “Major” Devices §  6-‐12 devices used for compaJbility tesJng

–  Vary models, specs, manufacturers, and networks §  AutomaJon less useful but cost-‐effecJve for key tests

§  “Market” Devices §  20-‐40 devices used for acceptance tesJng §  Primarily manual tesJng

6/2/15

19

Ques<ons?

If you have any further quesJons, please contact: Orasi Mobile [email protected]

Orasi Somware At A Glance

Our Company ·  Focus on enterprise soQware quality management

·  1800+ Customers ·  Partners with Mercury/HP since 1994

Our Technology ·  HP Test Management and AutomaJon ·  HP Performance TesJng ·  HP Mobile Center, NV ·  Perfecto Mobile, Mobile Labs

2012 HP Support Partner of Year 2013 HP Somware Partner of Year

Our Services ·  ImplementaJon Services ·  Install and Mentoring

·  Mobile, SAP, Security and Performance TesJng

·  Onsite, Rural or Offshore ·  SoQware Support

cloud-based, automated mobile app testing for the enterprise

Software