Stuart Reid PhD, FBCS (stuart@sta.co.kr)

© 2015 Stuart Reid

Making Your Reviews More Effective

Using the New ISO Standard (NIPA – 2nd July 2015)

• Context - ISO/IEC 20246 & the ISO 29119 standards

• The generic review process

• Individual review techniques

• Review types vs review attributes

• Current status

• Question?

Scope

Putting ISO 20246 in Context

ISO 29119/33063/20246 – Structure

BS 7925-1

BS 7925-2 IEEE 829

Concepts & Vocabulary

Part 1

Testing Techniques

Part 4

Documentation

Part 3 Part 2

Processes

Keyword-Driven Testing

Part 5

Process Assessment

ISO/IEC 33063

Reviews

ISO/IEC 20246 IEEE 1028

ISO 29119 Part 2: Testing Processes

TEST MANAGEMENT PROCESSES

ORGANIZATIONAL TEST PROCESS

DYNAMIC TEST PROCESSES

TEST MANAGEMENT PROCESSES

STATIC TEST

PROCESSES

ORGANIZATIONAL TEST PROCESS

DYNAMIC TEST

PROCESSES

Testing Processes – the intention

REVIEWS STATIC

ANALYSIS

V&V Taxonomy – ISO/DoD Washington, 2010

Verification & Validation

Formal Methods

Model Checking

Proof of Correctness

Specification-Based

Structure-Based

Experience-Based

V&V Analysis Simulation

Evaluation Quality Metrics

Model Verification

Static Analysis

Reviews

Static Testing

Dynamic Testing

Testing

ISO/IEC/IEEE 29119

TEST MANAGEMENT

PROCESSES

ORGANIZATIONAL TEST

PROCESS

DYNAMIC TEST

PROCESSES

Testing Processes – the end result

ISO/IEC/IEEE 20246

REVIEW PROCESS

ISO/IEC/IEEE 29119

TEST

MANAGEMENT

PROCESSES

ORGANIZATIONAL

TEST PROCESS

DYNAMIC TEST

PROCESSES

Testing Processes – the next step?

ISO/IEC/IEEE 20246

REVIEW PROCESS

ISO/IEC/IEEE ?????

STATIC ANALYSIS

ORGANIZATIONAL TEST PROCESS

TEST MANAGEMENT PROCESSES

TEST

PLANNING

TEST

MONITORING

& CONTROL

TEST

COMPLETION

ORGANIZATIONAL

TEST

DOCUMENTATION

FEEDBACK ON

ORGANIZATIONAL TEST

DOCUMENTATION

TEST PLAN UPDATES

TEST

PLAN

TEST

COMPLETION

REPORT

DYNAMIC TEST

PROCESSES

TEST

MANAGEMENT

PROCESSES

TEST PLAN,

TEST COMPLETION

REPORT,

TEST MEASURES

TEST

MEASURES

TEST PLAN,

CONTROL

DIRECTIVES

TEST PLAN,

CONTROL

DIRECTIVES

Overall Test Management Processes

WORK PRODUCT

REVIEW

PROCESS

REVIEW

MEASURES

TEST PLAN,

CONTROL

DIRECTIVES

Organise

Test Plan

Development

Identify &

Estimate Risks

Design Test

Strategy

Determine

Staffing and

Scheduling

Document

Test Plan

Schedule, Staffing

Profile

Test

Strategy

Estimated

Risks

Scope

Identify Risk

Mitigation

Approaches

Gain

Consensus on

Test Plan

Approved

Test Plan

Draft

Test Plan

Test

Plan Publish

Test Plan

Understand

Context

Treatment

Approaches

ISO 29119 Part 2 - Test Planning Process

Understand

Context

MUST INCLUDE

REVIEWS

ISO/IEC 20246 Work Product

Reviews

• Anything can be reviewed – in parts or the whole thing

• For example:

– Contracts – Requirements Specification – Design Specification – Code – Test Specification – User Manual – User Procedures – Project Plan – Test Plan – Development Plan

Why ‘Work Product’ Reviews?

• What do you think?

• Find defects

• Measure quality

• Educate reviewers

• Gain consensus

• Generate new ideas

• Motivate authors to improve their practices

Software Reviews – Main Purpose

Generic Review Process

Work Product Reviews - Generic Process

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

• Identify review scope – what is being reviewed – the purpose of the review (there may be more than one) – relevant supporting information, such as standards – the timeframes for the review

• Identify review characteristics – review activities – individual review techniques – checklists

• Distribute review materials (as early as possible) – product to be reviewed – baseline specifications – checklists

Planning & Preparation

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation & Analysis

Fixing &

Reporting

• Choose appropriate reviewers

– agreed with Project Manager

– assign specific roles

• Agree with each reviewer

– availability (and a substitute, just in case)

– the review role and focus

Planning & Preparation – Reviewers

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation & Analysis

Fixing &

Reporting

• Optional

• Review Leader

– presents the scope of the review to the review participants

– presents related documents to reviewers

– explains the role, responsibilities and focus of each review participant

– details the timeline for the review

• Reviewers (and author)

– formally commit to the review

Overview Meeting

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation & Analysis

Fixing &

Reporting

• Each reviewer identifies issues with the work product

• Identified issues shall be communicated to the review leader / author

Individual Review

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

• The author collates the Issues identified by the reviewers • Collated issues are analysed and a decision made on what

to do with each issue • Issue Review Status

– ‘rejected’ – ‘issue to be noted but no action’ – ‘issue to be addressed’ – ‘issue updated due to analysis’ – etc.

• Issues are then assigned based on their review status. – to authors – to other responsible individuals (e.g. to update earlier

specifications, organization standards)

Issue Collation & Analysis – 1 (of 2)

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation &

Analysis

Fixing &

Reporting

• Finally the review decision is made on the reviewed product:

– used as is

– updated based on the identified issues and used

– reworked and re-reviewed

– discarded

• The BIG Question – is this analysis and decision-making done by the author or done by a group of reviewers as part of a Review Meeting???

Issue Collation & Analysis – 2 (of 2)

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation &

Analysis

Fixing &

Reporting

• The author – addresses the outstanding issues – organizes for others to fix their outstanding issues in related

documents • in ‘earlier’ specifications • in organizational standards

– checks that all issues have been addressed – either:

• publishes the updated work product • submits the updated work product for re-review

• The reviewers – sign off on the final version (confirming that issues addressed)

• A Review Report is generated

Fixing & Reporting

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation &

Analysis

Fixing &

Reporting

Review Timelines

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation &

Analysis

Fixing &

Reporting

3 days

5-10 days

3 days

5-10 days

2 days

They vary!!

But need to be set and agreed.

Generic Review - Roles

Reviewer

Author

Planning &

Preparation

Overview

Meeting

Individual Review

Issue

Collation &

Analysis

Fixing &

Reporting

Review Leader

Reader Recorder/Scribe

Moderator

QA

SME

Customer

Management

Technical Lead

Individual Reviews

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

• Ad Hoc Reviewing

• Checklist-Based Reviewing

• Scenario-Based Reviewing

• Perspective-Based Reading (PBR)

Individual Review Techniques

• This is the most common approach (completely unstructured)

• Each reviewer is expected to find as many defects as possible of any type

• Little or no guidance on how to review is provided

• This approach is highly-dependent on reviewer skills

• Often the same issues are identified by different reviewers

Ad Hoc Individual Review

“60% of reviewers don’t prepare at all”

Individual Reviews -

Checklist-based Reviewing

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

• A systematic approach to identifying defects

• Typically review checklists take the form of a set of questions based on potential defects (and risk)

• Assign different reviewers to different checklists to – get wider coverage

– prevent the duplication inherent in the ad hoc approach

• There is a danger that some reviewers limit themselves to only considering the checklist entries

• Will normally take longer than using an ad hoc approach – often multiple passes are needed to cover all questions

(especially if checklist is long)

– Requires a systematic approach

Checklist-based Review

“50% OF REVIEWERS USE CHECKLISTS”

• Checklist defects are derived from experience – within the project – within the organization – across the industry as a whole

• Checklists should be specific to – the product under review (be wary of generic questions) – the methodology used to develop the work product (e.g. there may

be different checklist questions for requirements in the form of plain text to those in the form of use cases or user stories)

– the application domain of the work product (e. g. a checklist for a banking work product may be based on banking regulations while a checklist for an avionics work product would be based on avionics standards)

• Checklists should be based on risk – to ensure that the most important (and the most frequently-

occurring) risks are reviewed in more depth

Creating Checklists

• Many checklists are too long and never change

• The ideal checklist should be constrained to about 10 entries and regularly updated

– as entries become stale and find fewer issues (hopefully because the authors have learned and improved)

– new entries should reflect issues missed in the recent past

Managing Checklists

Individual Reviews –

Scenario-based Reviewing

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

• Used where multiple scenarios can be identified for different users

• Reviewers perform ‘dry runs’ on the work product to check functionality and handling of error conditions – following scenarios rather than reading the document from top to

bottom

• Assign different reviewers to different scenarios to: – get wider coverage – prevent the duplication inherent in the ad hoc approach

• There is a danger that some reviewers limit themselves to only considering the provided scenarios – and will miss defects of omission, where required functionality is not

included in the work product under review

• Research indicates that scenario-based reviews – can be more cost effective than checklist-based reviews – but they do take longer

Scenario-based Reviewing

• Where requirements, designs (or tests) are documented in a scenario-type format (e.g. use cases) then the existing specifications are a good starting point – the review can then use the modelled scenarios to drive the review

• However, scenarios typically also need to be created (we need multiple scenarios to cover everything): – overview scenario – most common scenarios – alternative scenarios – error condition scenarios – growth scenarios (e.g. for an architecture review) – input handling scenarios – output generating scenario – enquiry scenarios (getting specific results)

• Scenarios should be based on risk – to ensure that the most important (and the most frequently-used)

scenarios are reviewed in more depth

Identifying Scenarios

Individual Reviews -

Perspective-based Reading

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation & Analysis

Fixing &

Reporting

Perspectives and Roles

PRODUCT UNDER REVIEW

View 2

View 5

View 7

Perspective-based Reading (PBR)

Identify Stakeholders

Create/Identify PBR Scenarios

Perform Review Perform Review Perform Review Perform Review • Understand Stakeholder Perspective • Attempt to Create a Work Product • Use Checklist on the Work Product

Describes the stakeholder role

the reviewer is taking for this

review – and their interest in

the work product under review

Describes the high

level work product

that the stakeholder

would be expected to

develop

A checklist of questions

specific to the high-level

work product developed

Choosing your Review

• Milestone Reviews

• Inspections

• Technical Reviews

• Peer Reviews

• Walkthroughs

• Informal Reviews

• Author Check

• Buddy Check

• Pair Review

• Peer Deskcheck

Work Product Reviews – Types or Bespoke

Planning &

Preparation

Overview

Meeting

Individual

Review

Issue

Collation &

Analysis

Fixing &

Reporting

Chosen

Attributes

OR

• Purpose • Roles • Individual review techniques • Optional activities • Number of reviewers • Planned number of reviews • Work product type • Work product format • Formal reporting • Training required • Review improvement • Entry and exit criteria • Geographic distribution of reviewers

Work Product Reviews - Attributes

• Factors

– purpose

– product type

– product format

– risks

– reviewer availability

– reviewer skills (and availability of training)

– life cycle model

– budget (time and cost)

Choosing your Review

• Foreword • Introduction • 1 Scope • 2 Conformance • 3 Normative References • 4 Terms and Definitions • 5 Work Product Reviews - Introduction • 6 Software Review Process • 7 Review Techniques • Annex A (informative) Review Documentation • Annex B (informative) Review Roles • Annex C (informative) Review Attributes • Annex D (informative) Review Types (and attribute mapping) • Annex E (informative) Mapping to IEEE 1028 • Annex F (informative) Reviews - Life Cycle Mapping • Annex G (informative) Review Measurement & Improvement • Annex H (informative) Tool Support • Annex I (informative) Bibliography

ISO/IEC 20246 - Work Product Reviews - Contents

ISO/IEC 20246 Development

Working Draft (WD)

Committee Draft (CD)

Draft International Standard (DIS)

Final Draft International Standard (FDIS)

Final International Standard (FIS)

May

2014

May

2015

May

2016

WD1

DIS

FDIS

WD2

DIS2

ISO 29119/30363/20246 – Current Status

Testing Concepts & Vocabulary

29119 Part 1

Testing Techniques

29119 Part 4

Test Documentation

29119 Part 3 29119 Part 2

Keyword-Driven Testing

29119 Part 5

Process Assessment

ISO/IEC 33063

AUG ‘13

AUG ‘13 AUG ‘13 @ FDIS

@ DIS2 @ FDIS

Work Product Reviews

ISO/IEC 20246

DIS

Test Processes

• Context - ISO/IEC 20246 & the ISO 29119 standards

• The generic review process

• Individual review techniques

• Review types vs review attributes

• Current status

• Question?

Summary

Thank you for listening

Any Questions?

• Join ISO Working Group 26

– representing your national standards body

– 6 day meetings, every 6 months

– contribute between meetings

• Join a WG26 mirror group

– for your national standards body

• Contribute materials

• Review drafts

• Trial the standards on real projects

Do you want to be involved?

• stuart@sta.co.kr

– if you have any questions on the standards

– if you are interested in trialling the standard on a project, reviewing drafts or writing examples

• http://softwaretestingstandard.org/

– WG26 website

• http://www.jtc1-sc7.org/

– access to official documents released by WG 26

Finally…