software testing syntax-based testing. syntax coverage four structures for modeling software graphs...
Post on 21-Jan-2016
236 Views
Preview:
TRANSCRIPT
Software Testing
Syntax-based Testing
Syntax Coverage
Four Structures for Four Structures for Modeling SoftwareModeling Software
GraphGraphss
LogicLogic Input Input SpaceSpace
SyntaSyntaxx
Use Use casescases
SpecsSpecs
DesignDesign
SourceSource
Applied to
DNFDNFSpecsSpecs
FSMsFSMsSourcSourcee
Applied to
InputInput
ModelModelss
IntegInteg
SourcSourcee
Applied to
White-box testing
• white-box testing is the process of carefully and methodically reviewing the software design, architecture, or code for bugs without executing it.
• Unfortunately, white-box testing is rarely done in practice (unlike black-box testing).
Applying Syntax-based Testing to Programs
• Syntax-based criteria originated with programs and have been used most with programs
• BNF criteria are most commonly used to test compilers
• Mutation testing criteria are most commonly used for unit testing and integration testing of classes
5
Instantiating Grammar-Based Testing
Grammar-Based Testing
Program-based Integration Model-Based Input-Based
• Compiler testing
• Valid and invalid strings
Grammar
String mutation
• Program mutation
• Valid strings
• Mutants are not tests
• Must kill mutants
• Input validation testing
• XML and others
• Valid strings
Grammar• Test how classes interact
• Valid strings
• Mutants are not tests
• Must kill mutants
• Includes OO
String mutation
• FSMs
• Model checking
• Valid strings
• Traces are tests
String mutation
• Input validation
testing
• XML and others
• Invalid strings
• No ground strings
• Mutants are tests
String mutation
5.2
6
BNF Testing for Compilers (5.2.1)
• Testing compilers is very complicated –Millions of correct programs !
–Compilers must recognize and reject incorrect programs
• BNF criteria can be used to generate programs to test all language features that compilers must process
BNF Grammars
Stream ::= action*
action ::= actG | actB
actG ::= “G” s n
actB ::= “B” t n
s ::= digit1-3
t ::= digit1-3
n ::= digit2 “.” digit2 “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” |
“7” | “8” | “9”
Non-terminals
Terminals
Production rule
Start symbol
© Ammann & Offutt 8
Example
Ground StringGround String
G 22 08.01.90G 22 08.01.90
B 18 06.27.94B 18 06.27.94
Mutation OperatorsMutation Operators
• Exchange actG and actBExchange actG and actB
• Replace digits with all other digitsReplace digits with all other digits
Mutants using MOCMutants using MOC
BB 22 08.01.90 22 08.01.90
B 1B 199 06.27.94 06.27.94
Mutants using MPCMutants using MPC
BB 2222 08.01.90 08.01.90 GG 18 06.27.94 18 06.27.94
G G 112 08.01.90 2 08.01.90 BB 1 111 06.27.94 06.27.94
G G 332 08.01.90 2 08.01.90 BB 1 122 06.27.94 06.27.94
G G 442 08.01.90 2 08.01.90 BB 1 133 06.27.94 06.27.94
G G 552 08.01.90 2 08.01.90 BB 1 144 06.27.94 06.27.94
… … … …
Stream ::= action*action ::= actG | actBactG ::= “G” s nactB ::= “B” t ns ::= digit1-3
t ::= digit1-3
n ::= digit2 “.” digit2 “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” | “7” | “8” | “9”
Grammar
Mutation as Grammar-Based Testing
Grammar-based Testing
UnMutated Derivations
(valid strings)
Mutated Derivations
(invalid strings)
Grammar Mutation
(invalid strings)
Ground String Mutation
Invalid Strings Valid Strings
Generic coverage Generic coverage criteria can now criteria can now
be definedbe defined
Program-based Grammars• The original and most widely known application of
syntax-based testing is to modify programs• Operators modify a ground string (program under
test) to create mutant programs• Mutant programs must compile correctly (valid
strings)• Mutants are not tests, but used to find tests• Once mutants are defined, tests must be found to cause
mutants to fail when executed• This is called “killing mutants”
10
Mutation Testing
• Ground string: A string in the grammar– The term “ground” is used as a reference to algebraic ground
terms
• Mutation Operator : A rule that specifies syntactic variations of strings generated from a grammar
• Mutant : The result of one application of a mutation operator
– A mutant is a string
Mutant operators: Examples
Mutant operator In P In mutant
Variable replacement z=x*y+1; x=x*y+1;
z=x*x+1;
Relational operator replacement
if (x<y) if(x>y)
if(x<=y)
Off-by-1 z=x*y+1; z=x*(y+1)+1;
z=(x+1)*y+1;
Replacement by 0 z=x*y+1; z=0*y+1;
z=0;
Arithmetic operator replacement
z=x*y+1; z=x*y-1;
z=x+y-1;
What is program mutation?
P’ is known as a mutant of P.
There might be a test t in T such that P(t)≠P’(t). In this case we say that t distinguishes P’ from P. Or, that t has killed P’.
There might be not be any test t in T such that P(t)≠P’(t). In this case we say that T is unable to distinguish P and P’. Hence P’ is considered live in the test process.
Killing Mutants
• If mutation operators are designed well, the resulting tests will be very powerful
• Different operators must be defined for different programming languages and goals
• Testers can keep adding tests until all mutants have been killed
– Dead mutant : A test case has killed it
– Stillborn mutant : Syntactically illegal
– Trivial mutant : Almost every test can kill it
– Equivalent mutant : No test can kill it (equivalent to original program)
Given a mutant m M for a ground string program P and a test t, t is said to kill m if and only if the output of t on P is different from the output of t on m.
Error detection using mutation
Consider the following function foo that is required to return the sum of two integers x and y.
int foo(int x, y){return (x-y);}
Error detection using mutation
Now suppose that foo has been tested using a test set T that contains two tests:
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
First note that foo behaves perfectly fine on each test in, i.e. foo returns the expected value for each test case in T. Also, T is adequate with respect to all control and data flow based test adequacy criteria.
Error detection using mutation
Note that M1 is obtained by replacing the - operator by a + operator, M2 by replacing y by 0, and M3 by replacing x by 0.
int foo(int x, y){return (x+y);}
M1: int foo(int x, y){return (x-0);}
M2: int foo(int x, y){return (0+y);}
M3:
Let us evaluate the adequacy of T using mutation. Suppose that the following three mutants are generated from foo.
Error detection using mutation
Next we execute each mutant against tests in T until the mutant is distinguished or we have exhausted all tests. Here is what we get.
Test (t) foo(t) M1(t) M2(t) M3(t)
t1 1 1 1 0
t2 -1 -1 -1 0
Live Live Killed
T={ t1: <x=1, y=0>, t2: <x=-1, y=0>}
Questions About Mutation
• Should more than one operator be applied at the same time ?
– Should a mutated string contain one mutated element or several?
– Almost certainly not – multiple mutations can interfere with each other
– Extensive experience with program-based mutation indicates not
• Mutation operators exist for several languages– Several programming languages (Fortran, Lisp, Ada, C, C++,
Java)
– Specification languages (SMV, Z, Object-Z, algebraic specs)
– Modeling languages (Statecharts, activity diagrams)
– Input grammars (XML, SQL, HTML)
20
Mutation testingConsider the following program P
1.main(argc,argv)
2.int argc, r, i;
3.char *argv[];
4.{ r = 1;
5.for i = 2 to 3 do
6.if (atoi(argv[i]) > atoi(argv[r])) r = i;
7.printf(“Value of the rank is %d \n”, r);
8.exit(0); }
• Test Case 1:
input: 1 2 3
output: Value of the rank is 3• Test Case 2:
input: 1 2 1
output: Values of the rank is 2• Test Case 3:
input: 3 1 2
output: Value of the rank is 1
Mutant 1: Change line 5 to for i = 1 to 3 doMutant 2: Change line 6 to if (i > atoi(argv[r])) r = i;Mutant 3: Change line 6 to if (atoi(argv[i]) >= atoi(argv[r])) r = i;Mutant 4: Change line 6 to if (atoi(argv[r]) > atoi(argv[r])) r = i;Execute modified programs against the test suite, you will get the results:Mutants 1 & 3: Programs will pass the test suite, i.e., mutants 1 & 3 are not killableMutant 2: Program will fail test cases 2Mutant 1: Program will fail test case 1 and test cases 2Mutation score is 50%, assuming mutants 1 & 3 non-equivalent
Logic Mutation Testing Process1) Systematically create program variations (mutants) each
containing a single typical logic fault
if (a && b) if (a || b)
2) For each mutant, find an input that distinguishes the original program from the mutant program (killing)
- a && b and a || b evaluate to different truth values
Program-based Grammars
Original Method
int Min (int A, int B){ int minVal; minVal = A; if (B < A) { minVal = B; } return (minVal);} // end Min
With Embedded Mutants
int Min (int A, int B){ int minVal; minVal = A;∆ 1 minVal = B; if (B < A)∆ 2 if (B > A)∆ 3 if (B < minVal) { minVal = B;∆ 4 Bomb ();∆ 5 minVal = A;∆ 6 minVal = failOnZero (B); } return (minVal);} // end Min
6 mutants
Each represents a separate program
Replace one variable Replace one variable with anotherwith another
Changes operatorChanges operator
Immediate runtime Immediate runtime failure … if reachedfailure … if reached
Immediate runtime Immediate runtime failure if B==0 else failure if B==0 else does nothingdoes nothing
Syntax-Based Coverage Criteria
1) Strongly Killing Mutants:
Given a mutant m M for a program P and a test t, t is said to strongly kill m if and only if the output of t on P is different from the output of t on m
2) Weakly Killing Mutants:
Given a mutant m M that modifies a location l in a program P, and a test t, t is said to weakly kill m if and only if the state of the execution of P on t is different from the state of the execution of m immediately on t after l
Weak Mutation Example
• The complete test specification to kill mutant 1:
• (A = 5, B = 3) will weakly kill mutant 1, but not strongly
minVal = A;∆ 1 minVal = B; if (B < A) minVal = B;
• Mutant 1 in the Min( ) example is:
Equivalent Mutation Example
• Mutant 3 in the Min() example is equivalent:
minVal = A; if (B < A)∆ 3 if (B < minVal)
• The infection condition is “(B < A) != (B < minVal)”
• However, the previous statement was “minVal = A”– Substituing, we get: “(B < A) != (B < A)”
• Thus no input can kill this mutant
1 boolean isEven (int X)
2 {
3 if (X < 0)
4 X = 0 - X;
∆ 4 X = 0;
5 if (float) (X/2) == ((float) X) / 2.0
6 return (true);
7 else
8 return (false);
9 }
Strong Versus Weak Mutation
(X = -6) will kill mutant (X = -6) will kill mutant 4 under 4 under weak mutationweak mutation
27
Automated Automated stepssteps
Testing Programs with Mutation
Input test method
Prog Create mutants
Run T on P
Run mutants:• schema-based• weak• selective
Eliminate ineffective
TCs
Generate test cases
Run equivalence
detector
Threshold reached
?
Define threshold
no
P (T) correct
?
yesFix P
no
Mutation Operators for Java
Each occurrence of one of the arithmetic operators +, - ,*, / , and % is replaced by each of the other operators. In addition, each is replaced by the special mutation operators leftOp, and rightOp.
2. AOR –– Arithmetic Operator Replacement:
Each arithmetic expression (and subexpression) is modified by the functions abs(), negAbs(), and failOnZero().
1. ABS –– Absolute Value Insertion:
Each occurrence of one of the relational operators (<, ≤, >, ≥, =, ≠) is replaced by each of the other operators and by falseOp and trueOp.
3. ROR –– Relational Operator Replacement:
Mutation Operators for Java (2)
Each occurrence of one of the logical operators (and - &&, or - || , and with no conditional evaluation - &, or with no conditional evaluation - |, not equivalent - ^) is replaced by each of the other operators; in addition, each is replaced by falseOp, trueOp, leftOp, and rightOp.
4. COR –– Conditional Operator Replacement:
5. SOR –– Shift Operator Replacement:
Each occurrence of one of the shift operators <<, >>, and >>> is replaced by each of the other operators. In addition, each is replaced by the special mutation operator leftOp.
Each occurrence of one of the logical operators (bitwise and - &, bitwise or - |, exclusive or - ^) is replaced by each of the other operators; in addition, each is replaced by leftOp and rightOp.
6. LOR –– Logical Operator Replacement:
Mutation Operators for Java (3)
Each occurrence of one of the assignment operators (+=, -=, *=, /=, %=, &=, |=, ^=, <<=, >>=, >>>=) is replaced by each of the other operators.
7. ASR –– Assignment Operator Replacement:
8. UOI –– Unary Operator Insertion:
Each unary operator (arithmetic +, arithmetic -, conditional !, logical ~) is inserted in front of each expression of the correct type.
Each unary operator (arithmetic +, arithmetic -, conditional !, logical~) is deleted.
9. UOD –– Unary Operator Deletion:
Mutation Operators for Java (4)
Each variable reference is replaced by every other variable of the appropriate type that is declared in the current scope.
10. SVR –– Scalar Variable Replacement:
Binary search exampleint search ( int key, int [] elemArray){
int bottom = 0;int top = elemArray.length - 1;int mid;int result = -1;while ( bottom <= top ){
mid = (top + bottom) / 2;if (elemArray [mid] == key){ result = mid; return result;} // if partelse{ if (elemArray [mid] < key) bottom = mid + 1;
else top = mid - 1;}
} //while loopreturn result;
} // search
top related