A Theoretical Framework for Understanding Mutation-Based Testing Methods
Donghwan Shin and Doo-Hwan Bae
KAIST, South Korea
@ ICST 2016
Β© Donghwan Shin, 2016
Mutation-based testing has been widely studied for addressing various testing problems
Test set selection Fault localization Program repair
2
ππ ππ§
ππ₯
ππ¦
Artificially mutated programππ: an original programπ: a mutant generated from ππ
Artificial fault Partial fix
Systematically generate π from ππ
and use the differences between them
difference difference
difference
Β© Donghwan Shin, 2016
There is a solid theoretical framework for general testing process β βtesting systemβ
Such theoretical framework facilitates a clear understanding of the essence of complex problems
3
π· attempts to implement πΊ
π» is designed to consider πΊ and π·
πΆ determines the correctness of π· for π»
β―
ππ
ππ
π· is a set of programs πΊ is a set of specifications π» is a set of tests πΆ is a set of oracles
Fundamental testing factors and their relationships[1][2]
[1] J. S. Gourlay, βA mathematical framework for the investigation of testing,β Software Engineering, IEEE Transactions on, no. 6, pp. 686-709, 1983.[2] M. Staats, M. W. Whalen, and M. P. E. Heimdahl, βPrograms, tests, and oracles: the foundations of testing revisited,β in Proceedings of ICSE 2011, pp. 391-400.
Β© Donghwan Shin, 2016
Surprisingly little attention has been paid to the theoretical framework of mutation-based testing
4
Even the testing system is not adequate to mutation-based testing because it focuses on the correctness.
π· attempts to implement πΊ
π» is designed to consider πΊ and π·
πΆ determines the correctness of π· for π»
β―
ππ
ππ
πΊ is a set of specifications π· is a set of programs π» is a set of tests πΆ is a set of oracles
How to formally describethe behavioral differences between programs and its mutants in testing?
π
Β© Donghwan Shin, 2016
Key idea: a new testing factor for difference
5
π π‘, π = 1 π‘ππ’π ,0 ππππ π ,
if π is correct for π
otherwise
In the testing system, an oracle π implies the correctness of a program π for a test π as follows:
Let us define a new testing factor πΏto imply the difference between two programs for a test
π(π‘, ππ₯, ππ¦)
Β© Donghwan Shin, 2016
A theoretical framework for the difference in mutation-based testing
6
In this talk:β’ Define and extend a new testing factor to formalize the
βdifference-based testing frameworkβ
What you can do using this framework:β’ Formally describe the behavioral differences of programs
given a set of tests.
β’ Quantitatively represent and analyze the behavioral differences in a multi-dimensional space.
β’ Guide to understand mutation-based testing methods.
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
7
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
8
Β© Donghwan Shin, 2016
Test differentiator: a new testing factor (1/2)
9
Def. 1: A test differentiator π is a function such that
π π‘, ππ₯, ππ¦ = 1 π‘ππ’π ,0 ππππ π ,
if the behaviors of ππ and ππ are different for π
otherwise
π π‘, ππ, π = 1 βπ‘ detects the difference between ππ and πβ
Example
βThe test kills the mutantβ
Β© Donghwan Shin, 2016
Test differentiator: a new testing factor (2/2)
It clarifies tricky concepts in mutation-based testing.
10
A test π‘ detects a fault in ππ π π‘, ππ , ππ = 1
A test π‘ kills a mutant π π π‘, ππ, π = 1
ππ : the projection of the specification (i.e., true requirements of ππ)
It shows how π is important in mutation-based testing.
π observes internal values
π observes only output
Weak mutation Strong mutation
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
11
Β© Donghwan Shin, 2016
π-vector: extended π as a vector form
12
Def. 2: A π-vector π is an π-dimensional vector, such that
π π, ππ₯, ππ¦ = π π‘1, ππ₯, ππ¦ , β¦ , π(π‘π, ππ₯, ππ¦)
for a collection of tests π = π‘1, π‘2, β― , π‘π β π»π.
Test Behavior π-vector = behavioral difference
π ππ ππ π π π, ππ , ππ π π, ππ , π π π, ππ, π
π‘1 A A A
0,1,1,1 0,1,1,0 0,0,1,1π‘2 B A A
π‘3 C A D
π‘4 D A D
π-vector concisely describes the behavioral differences of programs for a set of tests
Example
Β© Donghwan Shin, 2016
π-vector = position in a multi-dimensional space
13
x
z
y
p(1,1,2)
A vector represents a point in a multi-dimensional space.
The position of a point is relative to the origin.
π π, ππ₯, ππ¦= π1, β¦ , ππ
We can think of a d-vector as the representation of a position in a space.
Β© Donghwan Shin, 2016
Position and its space from a π-vector
14
Def. 3: The position of a program ππ relative to another program ππ in a multi-dimensional space corresponding to a set of tests π is
π π, ππ , ππ₯ = ππππ ππ₯ .
The space is induced by π, ππ , π .
π = ππ, β― , ππ corresponds to the π-dimensions.
ππ corresponds to the origin of the space.
π corresponds to the notion of difference in the space.
A position of ππ relative to the origin ππ implies the behavioral difference between ππ and ππ for a set of tests π.
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
15
Β© Donghwan Shin, 2016
Position deviance relation (1/3)
Add more tests increasing dimensions make positions deviant from its origin
16
π = {β― , π‘π , π‘π+1, β― }
π©π± = β― , 0,0,β―
π©π² = β― , 0,1,β―
π = {β― , π‘π , β― }
π©π± = β― , 0,β―
π©π² is deviant from π©π± by π‘π+1
Example
Β© Donghwan Shin, 2016
Position deviance relation (2/3)
Forms a partial order based on transitivity.
17
π©π = β― , 0,0,β―
π©π = β― , 0,1,β―
π©π = β― , 1,1,β―
π©π = β― , 1,0,β―
π©π is more deviant than π©π.
π©π and π©π are incomparable.
Β© Donghwan Shin, 2016
Position deviance relation (3/3)
If ππ = ππ, the deviance relation on positions impliesthe dynamic subsumption* on mutants at the positions.
18
* Ammann, Paul, Marcio E. Delamaro, and Jeff Offutt. βEstablishing theoretical minimal sets of mutants.β ICST 2014.
Dynamic subsumptionIf ππ₯ is killed, then ππ¦ must be killed.
π©π± = β― , 0,0,β― = {ππ₯}
π©π² = β― , 0,1,β― = {ππ¦}
π©π¨ = 0,β― , 0 = {ππ}
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
19
Β© Donghwan Shin, 2016
Position Deviance Lattice (PDL) (1/4)
20
It shows the positions with their deviance relation.β’ Each dimension (=test) has only two positions, 0 or 1.
β’ The number of positions in a π-dimensional space = 2π.
π©0 = 0,0,0
π©2 = 0,1,0
π©5 = 1,0,1
π©7 = 1,1,1
π©1 = 1,0,0 π©3 = 0,0,1
π©4 = 1,1,0 π©6 = 0,1,1
π‘1 π‘2π‘3
π‘1 π‘2 π‘3
Dimensions
3-dimensional PDL
Example
Β© Donghwan Shin, 2016
Position Deviance Lattice (PDL) (2/4)
21
It growths as more tests added.
π©0
π©2
π©5
π©7
π©1 π©3
π©4 π©6
π‘1 π‘2π‘3
π‘1 π‘2 π‘3
Dimensions
Example
π©π, π©π
π©π, π©ππ©π, π©π
π©π, π©π
π‘1 π‘2
π©π, π©π, π©π, π©π
π©π, π©π, π©π, π©π
π‘1
π = {π‘1, π‘2} π = {π‘1, π‘2, π‘3}
π = {π‘1}
Β© Donghwan Shin, 2016
Outline
Test differentiator
β’ A new testing factor for the notion of difference
d-vectorβ’ Extended differentiator for a set of tests
Positionβ’ Redefined d-vector in a multi-dimensional space
Position deviance relation
β’ Formal relation between positions
Position Deviance Latticeβ’ Graphical model for positions and its deviance relation
Applications
22
Β© Donghwan Shin, 2016
Minimal set of mutants in PDL (1/2)
In PDL where ππ = ππ, mutants in βthe least deviantβ positions form the minimal set of mutants.
23
π©0 = {ππ}
π©2 = {}
π©5 = {π3}
π©7 = {π4}
π©1 = {π1} π©3 = {}
π©4 = {} π©6 = {π2}
π‘1 π‘2π‘3
Β© Donghwan Shin, 2016
Minimal set of mutants in PDL (2/2)
What is the maximum bound of the minimal set of mutants for π tests?β’ Key: if two mutants are at two comparable positions, then
one must be dynamically subsumed by another.
β’ Sperner's theorem says that the maximum number of incomparable nodes in an π-dim lattice is given as follows:
24
max(|ππππππππ|) =π
π/2
(E.g.) If we have 10 tests, # of minimal mutants β€ πππ
= 252
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
25
π (ππ, ππ,ππ) ππ ππ ππ ππ
ππ 1 1 0 0
ππ 0 0 1 1
ππ, π1, π2, π3, π4ππ = {}
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
26
ππ, π3, π4
π1, π2
π‘1
ππ = {π‘1}
π (ππ, ππ,ππ) ππ ππ ππ ππ
ππ 1 1 0 0
ππ 0 0 1 1
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
27
ππ
π3, π4π1, π2
π‘1 π‘2
(empty)
ππ = {π‘1, π‘2}
π (ππ, ππ,ππ) ππ ππ ππ ππ
ππ 1 1 0 0
ππ 0 0 1 1
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
28
ππ
π3, π4π1, π2
π‘1 π‘2
(empty)
ππ = {π‘1, π‘2}
Traditional mutation adequacy criterionβ’ A test suite that distinguishes the positions of mutants from
the position of ππwill likely detect real faults.
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
29
ππ
π3, π4π1, π2
π‘1 π‘2
(empty)
ππ = {π‘1, π‘2}
Claim for the existing mutation adequacy criterionβ’ In the βkilled areaβ, there are still several mutants which are
not distinguished in terms of their positions.
Β© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
30
ππ
π4
π1
(empty)
π2 (empty)
(empty) π3
π‘1 π‘2π‘3
Distinguishing more mutants increases fault detection*β’ A test suite that distinguishes the positions of mutants from
each other will likely detect real faults.
* Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. βDiversity-aware mutation adequacy criterion for improving fault detection capability.β Mutation 2016
Β© Donghwan Shin, 2016
Conclusion
Correctness-based think difference-based thinkβ’ Incorrect programs seem useless, while different programs
seems meaningful.
PDL may guide you to consider difference-based think.
31
π‘1 π‘2 π‘3
Dimensions