Regression Testing – What to Automate and how

Assurance with Intelligence

Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UKe: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173

In this session, I want to re-trace our early steps to

automationIs life hard at the end because we didn’t think it through the

beginning?

Assurance with Intelligence Slide 3

Automating regression tests• Regression tests are the most likely to be

stable and run repeatedly so:– Automation promises big savings of time– Automation ensures reliable execution and results

checking– Stable tests/software are usually easiest to

automate• Hold on! Paradox ALERT!!!!

– We regression test because things change– Chaotic, unstable environments need it the most– When things change automation is HARD!

Slide 4Slide 4 Assurance with Intelligence

What is the regression testing thought process?

Assurance with Intelligence Slide 5

How I think about regression testing• Establish what we mean by regression testing• Understand why regressions occur• Combating regression – there are choices• Who are the stakeholders for regression

testing and what do they want?• Context of regression testing• Regression test policy• Selecting what to regression test• Selecting what regression tests to automate

Assurance with Intelligence Slide 6

A common experience?• Regression testing is

a) Painfulb) Boringc) A waste of time

• “Let’s get a tool in”• Automating 10% of tests is easy• Automating 90% of tests is hard

(impossible?)• “Why are we doing this?”• “Let’s do it manually”

Assurance with Intelligence Slide 7

Let’s look at each thought process

Assurance with Intelligence Slide 8

Slide 9

When a software fault is fixed, it often happens that 'knock-on' effects occur

We need to check that when changing the faulty code, the developer has not introduced new faults

There is a 50% chance of introducing regression faults

Regression tests tell us whether new faults have been introduced- i.e. whether the system still works after a change to the

code or environment has been made. When environments are changed, we might also

regression test.

(What is...) regression testing?

Slide 9 Assurance with Intelligence

Slide 10

“Testing to ensure a change has not caused faults in unchanged parts of the system"

Not necessarily a separate stage Regression testing most important

during maintenance activities Regression tests are usually the first

to be considered for automation.

Regression testing 2

Slide 10 Assurance with Intelligence

How do regressions occur?• Something changes and

affects ‘working’ software• An enhancement is

implemented• A bug is fixed• Two bug fixes interfere

with each other• One programmer

overwrites another programmer’s fix (configuration management)

• Something in the environment setup

• Test data changes in some way

• A test changes in some way

• Test configuration/setup changes

• Etc. etc.

• Can you think of others?

Assurance with Intelligence Slide 11

Code fixes have a 50% chance of introducing side-effects in working

softwareAncient history – who cares whether it’s 10% or 90%?

Regressions have a disproportionate impact on effort, confidence, morale

Assurance with Intelligence Slide 12

Combating regression - choices• Several options:

– Don’t change anything!– Impact analysis (of code/data//test changes)– Regression testing (to detect unplanned changes)

• Impact analysis is tricky; not 100% reliable but...– Compare source code – see what’s changed– Module dependencies – see what’s affected– Rigour in managing environments and data– Rigour in managing source code

• Regression testing choices– Manual v automated– Test first or test last– All tests or a selection of tests– Others?

Assurance with Intelligence Slide 13

Who are our stakeholders?• Stakeholders are people interested in the outcome of test• What do they want? Ideally, nothing!• Sponsors, customers

– Regressions are an immediate setback/delay just when they thought you could move the project forward

• Users and testers– “All our good work is undermined!”

• Developers– They’ve missed something obvious, embarrassing; unplanned

work – could be trivial or substantial• Tech support

– “Users are going to be furious!”– “We feel really let down by dev, maintenance, testers”

Assurance with Intelligence Slide 14

Context of regression testing• Test- or behaviour-driven development:

– All tests are automated; these are our safety-net– Tests allow us to refactor with confidence

• Integration/System/Acceptance testing 1st pass– RT after main testing done – confirmation that all

tests pass with all fixes in place• Maintenance phase

– Minor fixes (don’t get impact analysed)– Enhancements fit with existing system– Refactoring/conversion/interface changes.

Assurance with Intelligence Slide 15

Regression test policy• Common:

– Business critical, high risk, error-prone areas

– FIFA – first in, first to automate?– LIFA – last in first to automate– Easily automatable

• But let’s step back a little– Focus on stakeholders– What do THEY actually want?

Assurance with Intelligence Slide 16

What would stakeholders want?• Sponsors, customers

– Want to avoid delays– Is the most critical functionality working?

• Tech support– Are we inconveniencing the users?– Is the most critical functionality working?

• Users and testers– Can we trust the results of our early (high priority tests)– (Maybe we aren’t so worried about the rest?)– Is the most critical functionality working?

• Developers– Want instant feedback (so they can fix with least hassle)– The best case for them doing their own automated tests!

Assurance with Intelligence Slide 17

If a developer ran your test team...• Whose problem is regression anyway?• It’s developers’ problem not the testers’!• Dev have been asked to fix/enhance/refactor

code without screwing up• How would a developer address the regression

problem?– Design a harness and code to execute transactions

at the API level– Easier, more maintainable, less sensitive to cosmetic

changes• A reusable asset that allows refactoring.

Assurance with Intelligence Slide 18

The real problem to solve?• If the problem is identifying change where there

shouldn’t be any...– With a few caveats, it doesn’t matter what tests we run

as long as we cover the important functionality – and that it behaves the same way as it did last time

– We still need to demonstrate the user interface isn’t adversely affected and the end-to-ends still work

– Is it POSSIBLE and BETTER to do this manually?– Why use a framework on top of a test execution tool on

top of a browser to test server-based code? Too complex!• Where does that leave automated test execution –

on the shelf? (It’s probably there already!)

Assurance with Intelligence Slide 19

Assurance with Intelligence Slide 20

Techniques for Testing GUI Applications• Eurostar 1997: the most popular paper

on my website is referenced by Wikipedia

• http://gerrardconsulting.com/sites/default/files/Techgui.pdf

• Won’t repeat the whole story here...• But most of the 1997 slides are still

relevant• Have we made NO progress since

1997?Assurance with Intelligence Slide 21

Selecting what regression tests to automate

1997

Assurance with Intelligence Slide 22

Manual v automatic scripts - very rough guideline

Manual Scripts Automatic Scripts Check list (application

conventions) Integration Synchronisation Client/server

communication Low-volume/complex

EP/BV

Check list (object states, menus, standard features)

Navigation Integration (for soak

tests) State transition Simple EP/BV and

decision tables.

What aspects to automate?• Pareto law:

– go for the easy scripts first– don’t waste time scripting low volume

complex scripts at the expense of high volume simple ones

• Consider hybrid approach:– use the tool to do navigation and data

entry– do the checking on-line or at end of test.

Automated test scripts• Coded scripts

– navigation through the system, some checklist test cases can be can be coded directly

– relatively easy to maintain• Recorded scripts

– automated scripts should be documented– then used to record the automated script– less easy to maintain

• Integrate all scripts into automated suite.

Migration of manual scripts• Automated scripts are easier to maintain

– if designed to be maintained– Means you have to build test automation

scaffolding before you script tests– Automation projects are software

development projects, not test projects• Allow for maintenance cost before

deciding to migrate functional scripts– delay migration until system is stable– soak and regression tests, as required.

Conclusion• The regression problem is a developers’

problem that can be solved by:1. Developer discipline2. Better impact analysis3. Automated testing at an API level4. Manual/automated testing (critical/E2E

level) for visibility to stakeholders• Is this what the big product

companies do?

Assurance with Intelligence Slide 27

Regression Testing – What to Automate and how

Assurance with Intelligence

Paul GerrardGerrard Consulting1 Old Forge CloseMaidenheadBerkshireSL6 2RD UKe: paul@gerrardconsulting.comw: http://gerrardconsulting.comt: 01628 639173