Download - Matthew Attaway, Perforce Software
![Page 1: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/1.jpg)
nt Session
Presented by:
Matthew
Brought to you by:
340 Corporate Way, Suite Orange Park, FL 32073 888‐2
T11 Concurre4/8/2014 2:00 PM
“Balancing Exploratory and Automated Testing in Agile”
Attaway
Perforce Software
300,68‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com
![Page 2: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/2.jpg)
Matthew Attaway Perforce Software
A jack of all trades and master of some, Matt Attaway has worked as a tester, developer, researcher, designer, manager, DevOps engineer, and elephant trainer. He currently manages the open source development group at Perforce Software, but prior to that he led a test team working on the highest profile projects at Perforce. When not tinkering with the latest software development pipeline tools, Matt enjoys experimenting with cocktails and playing games of all sorts—preferably simultaneously.
![Page 3: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/3.jpg)
2/18/2014
1
Balancing Exploratory and Automated Testing in Agile
Matt Attaway | @mataway
![Page 4: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/4.jpg)
2/18/2014
2
• Projects varied in size from 3-20 people
• Multiple languages, multiple stacks
My World
• GUIs, CLI tools, servers, and APIs
• Remote and co-located teams
• Scrum, Kanban, Scrumban, Waterscrumfallban
• Managed ~10 testers on 6 simultaneous projects
• Almost all "bet the company" projects
• Automation or bust!
My Constraints
• Exploratory testing a must
• Volatile projects; few design docs
• Extremely high bar for quality
• Executive demand for measurability and metrics
• Do more with less
![Page 5: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/5.jpg)
2/18/2014
3
Must handle the constant flood of change
My Process Requirements
• Must handle the constant flood of change
• Must integrate automated testing without sacficing exploratory testing
• Must communicate progress both to the product team and the executive teamteam and the executive team
Must not die
![Page 6: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/6.jpg)
2/18/2014
4
Ruthlessly optimize
Exploratory testing
T t l
Automated testing
Test plans
![Page 7: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/7.jpg)
2/18/2014
5
Exploratory testing
T t l
Automated testing
Test plans
Helm th on Moltke the Elder
“No plan survives contact with the enemy.”
–Helmuth von Moltke the Elder
![Page 8: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/8.jpg)
2/18/2014
6
Needed test plans per epic story feature and fix• Needed test plans per epic, story, feature and fix
• Easy report generation and APIs
• Easy access for testers
Cheap both in time and in money• Cheap, both in time and in money
Issue tracker
![Page 9: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/9.jpg)
2/18/2014
7
Focused around epic story feature and fix
Strengths
• Focused around epic, story, feature and fix
• Built-in report generation and APIs
• Testers already lived in it
Already installed and paid for!• Already installed and paid for!
Test plans highly visible to the developers
Bonuses
• Test plans highly visible to the developers
• Naturally supports multiple editors
• Extremely easy to extend
• Notifications of updates via email and RSS with diffs• Notifications of updates via email and RSS, with diffs
![Page 10: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/10.jpg)
2/18/2014
8
• Added a "Testing Notes" field to issues
Testers added test cases as they did their
Process
• Testers added test cases as they did their exploratory testing
• + used to flag completed test, - to flag pending test
• Anyone could add test cases to an issue
• Tweet sized test cases; should be clear for knowledgable tester
Example
- Attempt to create user with HTTP special characters- Attempt to create user with XML special characters+Attempt to create user that already exists
Example
![Page 11: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/11.jpg)
2/18/2014
9
• Devs handed off work by merging to mainline
Process & Measuring
• Issue not closed until all test cases complete
• Outstanding test work = issues in 'fixed' state
• Test work displayed on burndown charts; easy to spot when test was having a hard time keeping up
![Page 12: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/12.jpg)
2/18/2014
10
Automated testing
• Numerous automated test technologies and languages
CLI t t t l b t t t l d GUI t t t l• CLI test tools, web test tools, and GUI test tools
• Poorly written code; copy/paste gone wild
• Each product had one (or more!) test frameworks
• Too slow to add tests• Too slow to add tests
• Tests were dependent on each other
![Page 13: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/13.jpg)
2/18/2014
11
![Page 14: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/14.jpg)
2/18/2014
12
Given I am speaking at StarCanadaWhen I start my talkThen my heart rate will go up dramatically
• Forced code modularization
Why Cucumber?
• Test cases are legible and shareable
• Can drive any tool; can use most languages
• Tests are easy to add; frequently require no "real" code
• Works out of the box; huge development community
W it t t ti• Write tests anytime
• Everyone can automate
• Cucumber is, dare I say, fun
![Page 15: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/15.jpg)
2/18/2014
13
• Hour meeting at the beginning of the sprint; more as
Bugstorming
needed during sprint
• Wrote Gherkin as a team coming up with different attacks based on slotted stories
• Helped to share system knowledge and automated testing knowledge
• Track your hit rate; failed automated test is a bug
T k h t t t d h t d 't ( d h !)
Automated testing advice
• Track what you automate and what you don't (and why!)
• Aim to be able to add a test case in 15 minutes
• Always Reset State
• Automate all the things• Automate all the things
• Plan for parallelized tests from day one
![Page 16: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/16.jpg)
2/18/2014
14
Ruthlessly optimize
Avoid documents that do nothing(or make your documents do more)
![Page 17: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/17.jpg)
2/18/2014
15
Embrace your issue tracker
Detach writing automated tests from implementing automated tests
![Page 18: Matthew Attaway, Perforce Software](https://reader033.vdocuments.pub/reader033/viewer/2022042521/558de5681a28abe2428b46cc/html5/thumbnails/18.jpg)
2/18/2014
16
Regression tests == automated tests
The Cucumber Book by Aslak Hellesøy
Books
• The Cucumber Book by Aslak Hellesøy
• Cucumber & Cheese by Jeff Morgan
• Continuous Delivery by Jez Humble & David Farley
Exploratory Software Testing by James Whittaker• Exploratory Software Testing by James Whittaker