better test designs to drive test automation excellence

BT8 Concurrent Session 11/8/2012 2:15 PM

"Better Test Designs to Drive Test Automation Excellence"

Presented by:

Hans Buwalda LogiGear Corporation

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

Hans Buwalda LogiGear Corporation

An internationally recognized expert in testing, Hans Buwalda is the pioneer of keyword-driven test automation, an approach now widely used throughout the testing industry. Originally from the Netherlands, Hans is the CTO of California-based LogiGear, directing the development of the successful Action Based Testing™ methodology for keyword-driven test automation and its supporting TestArchitect™ toolset. Prior to joining LogiGear, he served as project director at CMG (now Logica) in Europe. Hans speaks frequently at international conferences on concepts such as Soap Opera Testing, Three Holy Grails of Test Development, Testing in the Cold, and Jungle Testing. Hans is coauthor of Integrated Test Design and Automation.

1

Presentation BT8

BETTER SOFTWARE EAST 2012

Better Test Designs toDrive Test Automation Excellence

Hans Buwalda, LogiGear

Mr. Playback

© 2012 LogiGear Corporation. All rights reserved.

Testing Under Pressure

DEADLINE

specification development test

© 2009 LogiGear Corporation. All rights reserved.© 2012 LogiGear Corporation. All rights reserved.© 2011 LogiGear Corporation. All rights reserved.© 2012 LogiGear Corporation. All rights reserved.

2

Testing Under Pressure

DEADLINE

specification development test

Develop tests in time:


Develop tests in time:• Test design• Auditing, acceptance• Preparations• Automation

Key Components for Success in Testing

• Appropriate Test design• Comprehensive automation architecture

• manageable, maintainable

• Management of the tests• tests and test scripts are products that need to be managed

• Management of the test process• managers want to know what is going on

• Documentation


• Documentation• Clear and useful reporting

• progress, results

• Quality Assurance• efficient and effective involvement of stake holders, users, auditors

3

Typical Problems with Automated Testing

• No clear direction

• Test design not well thought through

• Automation lacking architecturenot transparent and hard to manage

• Test process not well organized• test designers and test automators need/have different skill sets• stake holders don't know what is happening

T i d i d id d


• Testing underestimated or avoided• testing is difficult and expensive• it looks unattractive to spend money on testing• not considered something to think about

• Focus on tools and technology• engineers at the wheel

The 5% Rules of Test Automation

• No more than 5% of all test cases should be• No more than 5% of all test cases should be executed manually

• No more than 5% of all efforts around testing should involve automating the tests


testing should involve automating the tests

4

• Common in automation tools nowadays (but with different styles)

• Identify tasks for both test development and automation

• The test developer creates tests using actions

Keywords, Action Words

• The test developer creates tests using actions

• Each action consists of a keyword and arguments

• The automation task focuses on automating the actions

• Each action is automated only once

def action_NewProduct. . . .

def action_NewProduct. . . .

number name quantitynew product P-9009 Sledge Hammer 5

number name quantitynew product P-9009 Sledge Hammer 5


def action_AddQuantity. . . .

def action_AddQuantity. . . .

def action_CheckQuantity. . . .

def action_CheckQuantity. . . .

number quantityadd quantity P-9009 20add quantity P-9009 3add quantity P-9009 6

number quantitycheck quantity P-9009 34

number quantityadd quantity P-9009 20add quantity P-9009 3add quantity P-9009 6

number quantitycheck quantity P-9009 34

Keywords need a method

• By themselves keywords don't provide much scalabilitythey can even backfire and make automation more cumbersomea method can help tell you which keywords to use when, and how to organize the processthe process

• Today we'll look at Action Based Testing (ABT)addresses test management, test development and automationlarge focus on test design as the main driver for automation success

• Central deliveries in ABT are the "Test Modules"


developed in spreadsheetseach test module contains "test objectives" and "test cases"each test module is a separate (mini) project, each test module can involve different stake holders

5

Action Based Testing

Test Development Plan

Break down

Test Cases

TestObjectives

Test Module 1

Test Cases

TestObjectives

Test Module 2

Test Cases

TestObjectives

Test Module N

. . .

Automate


Actions

ACTION AUTOMATION

Automate

Example of a test module

• Consists of an (1) initial part, (2) test cases and (3) a final part• Focus is on readability, and a clear scope• Navigation details are avoided, unless they're meant to be tested


6

Example of a test module, "low level"

In "low level" tests the focus is typically on the user interaction with the UI (or other interactions with the system interfaces). The details of the interaction are visible in the test, since they are the target of the test. The "right level" of abstraction depends on the scope of the test, and is a matter of test design.

TEST MODULE Screen Flowuser

start system john

TEST CASE TC 01 Order Formwindow control

click main new orderwindow


windowcheck window exists new order

FINAL Exit applicationwindow

close application welcome

Test Design

• Effective test breakdown (into test modules)make sure every test module has a clear focusmake sure every test module has a clear focuskeep different kinds and levels of tests separate

• Right level of actionsas “high level” if possible, hiding as many details as much as possiblebut not if the details are relevant for the test


It is my believe that successful automation is not a technical challenge. It is most of all a test design challenge.

7

The Three “Holy Grails” of Test Design

• Metaphor to depict three main steps in test design

• Using "grail" to illustrate that there is no one perfect solution, but that it matters to pay attention (to search)

• About quality of tests, but also about scalability and maintainability, in particular in big projects

Organization of tests into Test Modules


Right approach for each Test Module

Proper level of detail in the test specification

What's the trick...


8

What's the trick...

• Have or acquire facilities to store and organize you content

• Edit your stuff

• Decide where to put whatassign and label the shelves

P t it th


• Put it there

• If the organization is not sufficient anymore, add to it or change it

Breakdown Criteria

• Straightforward CriteriaArchitecture of the system under test (client, server, protocol, sub systems, components, modules, ...)Functionality (customers, finances, management information, ...)y ( , , g , )Kind of test (navigation flow, negative tests, response time, ...)Ambition level (smoke test, regression, requirement based, aggressive, …)

• Additional CriteriaStakeholders

• accounting, compliance, Complexity of the test

• keep complex, hard to understand tests in separate modulesTechnical aspects of execution


Technical aspects of execution• special hardware, multi-station testing,

Overall project planning• availability of information, timelines, priorities, sprints,...

Risks involved• devote extra test modules to high risk areas (high complexity, high impact)

9

Properties of a good Breakdown

• Reflects the level of tests

Well differentiated and clear in scope• Well differentiated and clear in scope

• Balanced in size and amount

• Modules mutually independent


• Fitting the priorities and planning of the project

Example breakdown

• Tests of user interfacedoes function key F4 workdoes listbox xyz the right valuesis the tab order correct

• Form Tests do all the forms (dialogs screens pages) work:• Form Tests, do all the forms (dialogs, screens, pages) work:can data be entered and is it stored wellis displayed data correctsplit these from everything else

• Function tests, do individual functions workcan I count the orders

• Alternate paths in use casescan I cancel a transaction

• End-to-end testsdo all components of a system work well together in implementing the business processes


p y g p g plike enter sale order, then check inventory and accounting

• Tests with specific automation needslike multi station tests

• Tests of non-UI functions

• High ambition tests (aggressive tests)can I break the system under test

10

Example of an application under test

• Various item typestestsactionsinterface definitionsd t tdata setsfolders...

• Various operationsopencut, copy, pastecheck out...

• Various ways to initiate an operationcontext menu, with or without accelerator key


, ymain menu, with or without accelerator keytoolbarshort cut keyfunction keydrag and dropdouble click....

Defining some modules

• Test modules for operationsprimary and alternate pathsvarious values for fields like "comment" in check-inpaste in other projects

d t icopy and paste various groupsnot necessarily on each item type

• Test modules for itemsaddress all item types at least onceon each item type perform each operationnot necessarily each variant of each operation

• UI handlingtry for UI command if it starts the intended operation


y pnot necessarily on each item type or operation variant

• Concurrencytry concurrency sensitive operations with multiple stationsin varying concurrency scenarios, with and without local "refreshes"not necessarily each item type or operation variantcertainly not each UI command included

11

What is probably not a good design

• Navigational and functional tests are mixed"over checking": like a test for a insurance policy premium calculation also checks the existence of the window for data entry

• You have to change all of them for every new release of the system under test

• All test modules have a similar design


• Test modules are dependent on each other

• You can’t start developing any test modules early in the life cycle

Symptoms

• Tediousness in the test and test automation process

• No sense of control

• Complaining people

• Unnecessary high test maintenancechanges in the system under test impact many tests


g y p yhard to understand which tests need to be modified

• Difficulties in running any testteams start "debugging" tests

12

"Thou Shall Not Debug Tests..."

• Large and complex test projects can be hard to "get to run"

• If they are however start with taking a good look again atIf they are however, start with taking a good look again at your test design...

• Rule of thumb: don't debug tests. If tests don't run smoothly, make sure:

lower level tests have been successfully executed first -> UI flow in the AUT is stableactions and interface definitions have been tested sufficiently with their own


actions and interface definitions have been tested sufficiently with their own test modules -> automation can be trustedare you test modules not too long and complex?

Case Study

• Large IT provider

• New version of one of their major web-sitesNew version of one of their major web sites

• Test scope was user acceptance test (functional acceptance)

the users were the “business owners”

• Development was off-shore


13

Case Study

• Test development was done separate from automation

time line for test development: May Octtime-line for test development: May – Octtime-line for automation (roughly): Jan – Feb

• All tests were reviewed and approved by the business owners

acceptance was finished by the end of the test development cycle


Example of a Test Development PlanNr Module Business Owner Date to BO1 Portal Navigation, Audience Robyn Peterson 05 / 232 Portal Navigation, Search Ted Jones 05 / 273 Membership, registration Steve Shao 06 / 034 Portal Navigation, Category Ted Jones 06 / 085 Portal Navigation, Topic and Expert Ted Jones 06 / 136 Access Control Mike Soderfeldt 06 / 177 Portal Navigation, Task Ted Jones 06 / 228 Contact DSPP Ted Jones 06 / 279 Portal search Mike Soderfeldt 07 / 0110 Membership, review and update Steve Shao 07 / 0511 Program contact assignment Alan Lai 07 / 1112 Company, registration Steve Shao 07 / 1413 Catalog, view and query Robyn Peterson 07 / 1914 Site map Ted Jones 07 / 2515 Membership, affiliation Steve Shao 07 / 2816 Learn about DSPP Ted Jones 08 / 0117 Products and services Steve Shao, Robyn Peterson 08 / 0818 What's new Ted Jones 08 / 1119 C lif l St Sh Al L i 08 / 17


19 Company, life cycle Steve Shao, Alan Lai 08 / 1720 Specialized programs Ted Jones, Steve Shao 08 / 2221 Customer surveys Ted Jones 08 / 2922 Software downloads Mike Soderfeldt 09 / 0123 Newsletters Ted Jones 09 / 0624 Internationalization and localization Ted Jones 09 / 1325 Membership, life cycles Steve Shao 09 / 1926 Collaboration, forums Ted Jones 09 / 2327 Collaboration, blogs Mike Soderfeldt 09 / 2828 Collaboration, mailing lists Ted Jones 10 / 03

14

Review Process with Stake Holders

Test Team sends draft Stake Holder reviews:

START

Test Team sends draft Module to Stake Holder - coverage

- correctness

changes needed? Stake Holder returns notice of approval

no

yes


Stake Holder returns notes:

- additions- corrections

Test Team receives and processes notes

Test Team marks the Module as "Final"

END

Case Study, Results

• All tests were developed and reviewed on schedulemany notes and questions during test development phase

• The automation was 100% of the testsall actions were automated, thus automating all test modules

• The test development took an estimated 18 person monthsone on-shore resource, two off-shore resources

Th t ti t k b t d t th


• The automation took between one and two monthsfocused on actionsmost time was spent in handling changes in the interface (layout of pages etc)

15

Grail 2: Approach per Test Module

• Organize the test design process around the test modules

• Plan the test module:when to develop: is enough specification availablep g pwhen to execute: make sure the functionality at action level is well-tested and working already

• Process:analysis of requirementsformulation of "test objectives"create "test cases"

• Identify stakeholders and their involvement:users subject matter experts


users, subject matter expertsdevelopersauditors

• Choose testing techniques if applicable:boundary analysis, decision tables, transition diagrams, soap opera testing, ...

Eye on the ball, Scope

• Always know the scope of the test module

Th h ld b bi• The scope should be unambiguous

• The scope determines many things:what the test objectives arewhich test cases to expectwhat level of actions to usewhat the checks are about and which events should generate a


what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)

16

Make explicit test objectives

...TO-3.51 The exit date must be after the entry date...

test objective TO-3.51

name entry date exit dateenter employment Bill Goodfellow 2013-10-02 2013-10-01


enter employment Bill Goodfellow 2013-10-02 2013-10-01check error message The exit date must be after the entry date.

Grail 3: Specification Level, choosing actions

• Scope of the test determines the specification level

• As high level as appropriate, as little arguments as iblpossible

Use default values for non-relevant arguments

• Clear names (usually verb + noun usually works well)to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify client" (or vice versa)tests are not C++ code: avoid "technical habits", like mixed case and (worse) underlines


• Manage the Actions

• Document the Actions

• By-product of the test design

17

Grail 3: Example of using actions

section Capture initial numberkey

key navigate F7

In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test

key navigate F7key navigate 3

page tablocate page tab Scan Criteria

windowwait for controls loaded search

textcheck breadcrumb # search breadcrumb

window control valueselect search scan direction Backward

general functions > search


window control valueenter value search business date match # bus date

source controlclick search go

windowwait for controls loaded search results

store ascapture sequence num >> seq num

Grail 3: Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test

variable

get sequence number >> seq num


18

Mid-level actions

• Most tests will have low level and high level actionslow level: generic operations, know the interface, don't know the functionality

• examples: "selection menu item", "expand tree node", ...high level: business oriented operations know the functionality don't know thehigh level: business oriented operations, know the functionality, don t know the interface

• examples: "enter purchase order", "check inventory of article"

• For systems with many screens and fields, consider an in-between layer:

mid level: screen oriented, know the interface, know bit of functionality as well


• Examples of mid-level actions: "assign all address fields" "check all address fields"

Test Design and Agile

• Keywords are suitable for agile projects:tests are easier to create and understand, in particular for non-programmersthey allow test development without a need for details that haven't been defined yetdefined yetautomated tests can quickly follow changes in the system under test

• Action Based Testing in itself is quite agilefocused on products and cooperationflexible in process, in fact each test module can have its own processtest modules are usually very suitable to drive system development


• Test design will need to find its place in agile projectsidentifying test modulesin larger scale projects this may require at least some overall test planning

19

Test Development and Automation in sprints

product owner team prod owner

& team

Test ModuleDefinition(optional)

Test Module Development

Interface Definition

Sprint ProductsProduct Backlog

Test re-use

Automation re-use

owner & team


Interface Definition

Action Automation

Test Execution

Summary

• Keywords is one of the techniques for automated testing, in addition to record & playback and scriptingscripting

• In itself keywords are not a silver bullet, it needs a good approach, careful planning and good organization to be successful

T t d i t t h l i ll th


• Test design, not technology, is usually the dominating success factor for automation success

20

ReferencesAction Based Testing, Hans Buwalda, Better Software, March 2011

Action Figures (on model-based testing), Hans Buwalda, Better Software, March 2003

Automating Software Testing, Dorothy Graham, Mark Fewster, Addison Wesley, 1999

Experiences of Test Automation, Dorothy Graham, Mark Fewster, Addison Wesley, 2012

Happy About Global Software Test Automation, Hung Nguyen, Michael Hackett et al, Happy About, 2006

Integrated Test Design & Automation, Hans Buwalda et al, Addison Wesley, 2002

Lessons Learned in Software Testing, Cem Kaner, James Bach, Bret Pettichord, Wiley 2002

QA All Stars, Building Your Dream Team, Hans Buwalda, Better Software, September 2006

Soap Opera Testing, Hans Buwalda, Better Software Magazine, February 2005


Testing Computer Software, Cem Kaner, Hung Nguyen, Jack Falk, Wiley, 1999

Testing with Action Words, Abandoning Record and Playback, Hans Buwalda, Eurostar 1996

The 5% Solutions, Hans Buwalda, Software Test & Performance Magazine, September 2006

better test designs to drive test automation excellence

Technology