Liang Gaoliangg@gmail.com

E.T. 没有测试用例？ Bug越测越多？ 测试不可复现？ 如何积累？

Agenda Definition of E.T. and it is difference

between “Scripted Testing) How to do a good E.T. E.T testing management

Definition of E.T. Exploratory testing is simultaneous learning, test design, and test

execution

"Exploratory testing involves simultaneously learning, planning, running tests, and reporting / troubleshooting results."

Dr. Cem Kaner (2001)

"Exploratory testing is an interactive process of concurrent product exploration, test design and test execution.”

” To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory

testing.”

James Bach, Satisfice (2001)

No E.T? It is Everywhere 场景：

首轮测试之前对特性不熟悉，首轮测试用于用例熟悉，没有 Bug

次论涌现出许多测试想法，开始发散 /冒险 /即兴 , 测出 Bug

按照此方法，越测越有感觉， Bug越多，总不收敛软件发布日期推迟

场景：规定测试人员每周必须发现的 Bug的数量和质量测试人员：“照着测试用例测肯定找不到 Bug”，开始发散 /冒险 /即兴

Different Between E.T and Scripted

Product Product

E.T.

Bug

Review

Release

Test Plan

Test Strategy

Test Plan

Review

Manual Execution

Bug

Release

Adapting

Difference between E.T. and Scripted

E.T.’s goal is to find Bug Scripted testing is to measure coverage,

and be comprehensive

FullyScriptedTesting

Ad-hocTesting

AutomatedTests

BugHunting

Exploratory Testing

Agile software development is not conventional software development done more quickly or done on tippie-toe. Agile software development is software done differently.

Ron Jeffries, (e-mail on agile-testing list, April 24, 2002)

•proven (no single technique is new)•application oriented•planned and disciplined•controllable and reliable•risk minimizing•Two sides of extreme programming:

• for the developer: freedom, flexibility, fun• for the manager: controllability, reliability, high

quality

Extreme Programming

Agile software testing is not conventional (scripted) software testing done more quickly or done on tippie-toe.

Exploratory Testing:•proven (no single technique is new)•application oriented•planned and disciplined•controllable and reliable•risk minimizing•Two sides of Exploratory Testing:

• for the tester: freedom, flexibility, fun• for the manager: controllability, reliability, high

quality

Exploratory Testing

Agenda Definition of E.T. and it is difference

between “Scripted Testing) How to do a good E.T. E.T testing management

Let’s watch some Video first

Play exploratory testing demo video

Make sure the right attitude

Experiential and intuitive approaches:Develop a sense of smell“We have good hunches about where the bugs

are lurking.”“Let’s jump in an explore the system’s behavior

hands-on, so we can decide how to test it.”“Let’s find the important bugs fast, and worry

about the test paperwork later.”

Doing Exploratory Testing

Keep your mission clearly in mind. Keep notes that help you report what you

did, why you did it, and support your assessment of product quality.

Keep track of questions and issues raised in your exploration.

To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.

Taking Notes Test Coverage Outline/Matrix Evaluation Notes Risk/Strategy List Test Execution Log Issues, Questions & Anomalies

It would be easier to test if you changed/added…

How does … work?Is this important to test? How should I test it?I saw something strange…

Approaches / Styles of ET

At the heart of all ET styles:Questions and Questioning Skills

Characterize the styles with respect to each other:Do they focus on:

○ Method of questioning?○ Method of describing or analysing the product?○ The details of the product?○ The patterns of use of the product?○ The environment in which the product is run?

To what extent would this style benefit from group interaction?

Heuristic Risk Based Testing

The purpose of finding errorsMake a prioritised list of risks.Perform testing that explores

each riskAs risks evaporate and new

ones emerge, adjust your test effort to stay focused on the current crop.

Agenda Definition of E.T. and it is difference

between “Scripted Testing) How to do a good E.T. E.T testing management

E.T. is good for Provide rapid feedback on a new product or

feature Learn the product quickly You already tested using scripts, and seek

to diversify the testing Want to find the single most important bug

during the shortest time Want to investigate the status of a particular

risk Improving on scripted tests

E.T. in Session

1 pair (2 engineers) of each team Call it the “Extreme testing”, or “Bug

Sprinting” to energize our tester Rank the outcome if you want Make sure the top performer gets rewarded

Short: 60 minutes (+-15)Normal: 90 minutes (+-

15)Long: 120 minutes (+-15)

E.T. in Session – Tools you can use

Session Tester

Reviewable Result: The session sheet

Charter#AREAS

Start Time Tester Name(s) Breakdown

#DURATION#TEST DESIGN AND EXECUTION#BUG INVESTIGATION AND REPORTING#SESSION SETUP#CHARTER/OPPORTUNITY

Data Files

Controlled E.T. Make testing fun Improve test cases and test plans

Controlled E.T.代码库 开发 测试

反馈意见

手工第一轮

手工第二轮

手工第三轮

手工第四轮

开发新特性

允许新特性代码 Check in

发布日期

只允许 Bug 修复 Check in

只允许重大 Bug修复 Check in,中等以下 Bug不修复，会放在

Release note中

Controlled E.T.

Scripted Testing

Scripted Testing

Scripted Testing

E.T.

E.T.

E.T.

Test Plan Update Test Plan Update Test Plan Update

Round 1 Round 2 Round 3

Controlled E.T. Test team’s resource consumption per

week:25% of the group’s time developing new tests50% executing old tests (including bug

regression)25% on exploratory testing

Lessons learned...(1) Pair testing

Speeded up reporting errors Excellent to promote creativity Encouraged discussion Demanded a “strong voice” to keep everyone focused! Enables effective testing of short-lived, intense projects... but has also proved very useful on many test projects,

also larger (i.e. at Microsoft, ref. J.Bach)

Lessons Learned...(2) Exploratory testing

Requires skilled testers Requires some pre-set restraints, and thorough planning Requires a system for briefing and reporting/ documenting (ref.

J.Bach - “Session based test management”) Requires guidance from test manager ..but when the requirements are fulfilled, it proved to be very

successful. Easy to motivate MANY deviations found …and it gave testers a greater feeling of participation