Run smoke test on AWS DeviceFarm@naokiainoya

About me• Naoki Ainoya(@naokiainoya)

• iOS/Server/Infra engineer

• Swift/Scala/Go/Docker...

• Android...?

• Recruit Marketing Partners Co, Ltd.

• Works for 英単語サプリ mainly

Support android development

• Maintain Jenkins CI system for building android

• Build apk with using Docker/Walter

• Ref: http://ainoya.io/docker-android-walter

Smoke testing• e.g. smoke test may ask basic

questions like "Does the program run?"

• Smoke testing performed on a particular build is also known as a build verification test

• Ref: http://www.slideshare.net/confiz/software-testing-methods-levels-and-types

How many test do I have to test?• There are a lot of environments!

• Different Screen resolution

• Different Architecture

• arm, x86, ...

• We want to test on as many devices as possible

But we don't have such a many devices

Actually even I want to use

AWS DeviceFarm!• Device Farm is an app testing

service that enables you to test your Android, iOS, and Fire OS apps on real, physical phones and tablets that are hosted by Amazon Web Services (AWS).

• Ref: http://www.slideshare.net/AmazonWebServicesJapan/20150722-black-belttechwebinarawsdevice-farm

• Ref: http://docs.aws.amazon.com/devicefarm/latest/developerguide/welcome.html

How does AWS DeviceFarm work

• create project

• schedule runs

• select device pools

• upload your '.apk'

• specify test types

• JUnit, Espresso, Robotium etc..

• Upload test package separeted from '.apk'

• view reports

Upload .apk to DeviceFarm• Once uploading ".apk" , DeviceFarm detects

configuration automatically

Select devices• You need to select/create "device pools" which

you want to test

Supported devices• Show device list with "aws-cli": "aws

devicefarm list-devices | jq '.devices |map(select(.platform=="ANDROID")|.model)[]' | wc -l"

over 103 devices!

Supported devices• Command result: "aws devicefarm list-devices |

jq '.devices |map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"

Supported devices• Command result: "aws devicefarm list-devices | jq '.devices |

map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"

• "DROID Turbo" "DROID Ultra" "G Flex2" "G Pad 7.0"

• "G2" "G3" "G4" "Galaxy Light"

• "Galaxy Note 2" "Galaxy Note 3" "Galaxy Note 4" "Galaxy Note 5"

• "Galaxy Note" "Galaxy S3 LTE" "Galaxy S3 Mini" "Galaxy S3"

• "Galaxy S4 Active" "Galaxy S4 mini" "Galaxy S4" "Galaxy S5 Active"

• "Galaxy S5" "Galaxy S6 Edge" "Galaxy S6 Edge+" "Galaxy S6"

Supported devices• Command result: "aws devicefarm list-devices | jq '.devices

map(select(.platform=="ANDROID")|.model)[]' | sort | uniq"

• "Galaxy Tab 2 10.1" "Galaxy Tab 3 10.1" "Galaxy Tab 3 7.0" "Galaxy Tab 3 Lite 7.0"

• "Galaxy Tab 4 10.1 Nook" "Galaxy Tab 4 7.0 Nook" "Kindle Fire HD 6" "Kindle Fire HDX 7"

• "Moto E - 2nd Gen" "Moto G - 2nd Gen" "Moto G" "Nexus 10"

• "Nexus 4" "Nexus 5" "Nexus 6" "Nexus 7 - 1st Gen"

• "Nexus 7 - 2nd Gen" "One M7" "One M8" "Optimus Fuel"

• "Optimus G Pro" "Optimus L70" "Optimus L90" "Xperia Z2"

• "Xperia Z3 Compact" "Xperia Z3"

Supported devices

But it does't have a lot of Japanese popular devices

Supported Test frameworks• Appium, Calabash

• JUnit, Espresso, Robotium etc…

Test reports an result summary

Test reports an result with screen shots of device

Run smoke test with "Built-in Fuzz Test"

• Same kind of monkey test

Integrate smoke-test into CI

Jenkins AWS DeviceFarm

Developer Slack

Integrate smoke-test into CI

GHE

Jenkins AWS DeviceFarmDeveloper

Slack

push codes

webhook

Build & upload apk

Fetch Test results

Post results

Receive results

Slack integration• At First, Jenkins DeviceFarm plugin was good

to me…

Slack integrationIt has option allows to store all test results

(Screenshot, monkey-test.log, .logcat)

Slack integration• But Jenkins DeviceFarm plugin didn't work

well.. (Failed to store the test results)

• Finally I used "jedi4ever/devicefarm-cli" tool written in Go

Use AWS DeviceFarm API• e.g. "aws-­‐sdk-­‐go":  

• "func  (*DeviceFarm)  CreateUpload"  

• Uploads an app or test scripts.

• "func  (*DeviceFarm)  ScheduleRun"  

• Schedules a run.

• "func  (*DeviceFarm)  ListArtifacts"  

• Get information about artifacts.

• An artifacts includes a s3 URL of its contents.

• Ref: https://godoc.org/github.com/aws/aws-sdk-go/service/devicefarm

Slack integration example

Round up• AWS DeviceFarm makes tests easily

• On many real/physical devices

• But it does't have a lot of Japanese popular devices

• Try also SauceLabs for comparison :)

• I want to use STF with many devices