occf: a framework for developing test coverage measurement tools supporting multiple programming...
DESCRIPTION
http://www.icst.lu/ I talked about this slide at ICST 2013 Testing Tools Track.TRANSCRIPT
ICST Testing Tools Track 2013
2013/03/20
Kazunori Sakamoto, Kiyofumi Shimojo, Ryohei Takasawa, Hironori Washizaki, Yoshiaki Fukazawa
Waseda University
Dept. Computer Science and Engineering
Motivating Example
• Coverage based tools have differences
2013/3/20 ICST Testing Tools Track 2013 2
int min(int a, int b) {
return a < b ? a : b
}
@assert void testMin() {
assertEquals(1, min(1, 2));
}
• Many tools say statement coverage is
100% (1/1)
• EMMA says line coverage
is 86% (6/7)
Java
JUnit
• Tools use different measurement units
Comparison of 25 coverage tools
2013/3/20 ICST Testing Tools Track 2013 3
Tool Mecha nism
Hierar chical view
Free State ment
Deci sion
Cond ition
Condit ion/D ecision
Lang uages
Cobertura Binary ✔ ✔ ✔ ✔ Java
EMMA Binary ✔ ✔ ✔ ✔ Java
JCover Code ✔ ✔ ✔ Java, Groovy
Clover Code ✔ ✔ ✔ Java
Agitar Binary ✔ ✔ ✔ Java
OpenCover Processor ✔ ✔ ✔ ✔ C#
NCover Source ✔ ✔ ✔ C#
dotCover Binary ✔ ✔ C#
gcov Compiler ✔ ✔ ✔ C++
COVTOOL Code ✔ ✔ C++
Comparison of 25 coverage tools
2013/3/20 ICST Testing Tools Track 2013 4
Tool Mecha nism
Hierar chical view
Free Statement
Decision
Condition
Condit ion/D ecision
Lang uages
Bullseye Code ✔ ✔ ✔ ✔ ✔ C++
Intel Code Coverage Tool
Compiler ✔ ✔ Java
Squish Coco Code ✔ ✔ ✔ ✔ ✔ Java
TCAT Code ✔ ✔ C++, Java
Parasoft Test
Code ✔ ✔ ✔ ✔ ✔ C++, C# Java
PurifyPlus Binary ✔ ✔ ✔ ✔ ✔ C++, C# Java
Semantic Designs
Code ✔ ✔ ✔ C++, C# Java, etc.
Coverage Validator
Code ✔ ✔ ✔ C++, C# Java, etc.
Comparison of 25 coverage tools
2013/3/20 ICST Testing Tools Track 2013 5
Tool Mecha nism
Hierar chical view
Free Statement
Decision
Condition
Condit ion/D ecision
Lang uages
ScriptCover Code ✔ ✔ JavaScript
Coverage.py Processor ✔ ✔ ✔ Python
rcov Processor ✔ ✔ ✔ Ruby
SimpleCov Processor ✔ ✔ ✔ Ruby
Devel::Cover Processor ✔ ✔ ✔ ✔ ✔ Perl
xdebug Code ✔ ✔ ✔ PHP
LuaCov Processor ✔ ✔ Lua
• Tools support different criteria and langs
• Differences between tools cause problems (*) *) Rüdiger Lincke, Jonas Lundberg and Welf Löwe: “Comparing Software Metrics Tools,” ISSTA '08 Proceedings of the 2008 international symposium on Software testing and analysis, pp. 131-142, 2008.
Problems and tool solutions
• Differences between tools
– Hard to measure coverage of software using multiple langs such as Web apps
– OCCF aids to provide consistent criteria
• High development costs
– Hard to develop coverage tools supporting new programming languages
– OCCF aids to develop new coverage tools
2013/3/20 ICST Testing Tools Track 2013 6
7
int func(int a) { if (branch(2, a == 0)) { stmt(0); printf("a == 0"); } else if (branch(3,false)){ stmt(1); printf("a != 0"); } }
• Inserts automatically • Outputs coverage info • Has no side effect • Measures completely
int func(int a) { if (a == 0) { printf("a == 0"); } else if (false) { printf("a != 0"); } }
Statement and
branch coverage
Function
printf("a == 0") printf("a != 0")
Statement
stmt (0) stmt (1)
Statement
Statement
Statement
int func(int a)
Inserts into AST
Abstraction syntax tree(AST)
void stmt(int id) { RECORD_STATEMENT(id); }
Measurement approach
2013/3/20 ICST Testing Tools Track 2013
OCCF
Architecture
2013/3/20 ICST Testing Tools Track 2013 8
AST Generator
C Java
AST Transformer
OCCF Core
C Java
Coverage Report
Tool
Fault Localization
Tool
Test Case Minimization
Tool
Python
Python Parsers (Code2Xml)
Cold spot
Trans formers
Tools
Process and design
2013/3/20 ICST Testing Tools Track 2013 9
OC
CF
Testin
g F
ram
ew
ork
~~~~ ~~~~ ~~~~
~~~~ ~~~~ ~~~~
Production code
Test code
Covera
ge
Rep
ort
Tool
Fau
lt Localiz
atio
n
Tool
Test C
ase
Min
imiz
atio
n
Tool
0101 0001 1110 1010
coverage data
~~~~ ~~~~ ~~~~ ~~~~
~~~~ ~~~~ ~~~~ ~~~~
Pass Fail Pass Fail
Test Results
Instrumentation code
Modified Production
code
Modified Test code Instrumentation
code
library
Instrumentation library
Process and design in detail
2013/3/20 ICST Testing Tools Track 2013 10
AST G
en
era
tor
AST F
inder
AST In
serte
r
AST AST AST
source code
~~~~ ~~~~ ~~~~
Cod
e G
en
era
tor
source code
~~~~ ~~~~ ~~~~ ~~~~
Java
C
Python
Instrumentation code
XML object in .NET Framework
Implementation
• OCCF is developed in C# – Works on .NET Framework and Mono
• AST Generator utilizes existing parser libraries • ANTLR for C, C#, Java, JavaScript, Lua
– http://www.antlr.org/ – Reuse existing grammar files on the repository
• Standard library (parser module) for Python 2, 3 – http://docs.python.org
• ruby_parser for Ruby 1.8, 1.9 – https://github.com/seattlerb/ruby_parser
• srcML for C++ – http://www.sdml.info/projects/srcml/
2013/3/20 ICST Testing Tools Track 2013 11
Demo
• Insert instrumentation code in eight languages: C, C++, C#, Java, JavaScript, Python, Ruby, Lua
• Let me show four materials in Java
– Original code
– Original AST in XML
– Modified AST in XML
– Modified code
2013/3/20 ICST Testing Tools Track 2013 12
Hot spot in OCCF
• Finding nodes to be inserted – Providing enhancements for LINQ to XML
e.g. root.Descendants("statement")
.Select(e => e.FirstElement())
• Instrumentation library – Recording statements and branches
• Text file, Binary file, Shared memory, TCP/IP, …
– Basically, implemented in each language
• Function call for instrumentation library – Function calls are inserted into each langs
2013/3/20 ICST Testing Tools Track 2013 13
Finding statements in C#
2013/3/20 ICST Testing Tools Track 2013 14
statement : declaration_statement | labeled_statement | embedded_statement ; embedded_statement : block | selection_statement // if, switch | … ; Grammar
var decls = root.Descendants("declaration_statement"); var stmts = root.Descendants("embedded_statement") .Where(e => e.FirstElement().Name() != "block"); return stmts.Concat(decls); Finder
Finding statements in JavaScript
2013/3/20 ICST Testing Tools Track 2013 15
statement : statementBlock | variableStatement | emptyStatement | expressionStatement | ifStatement | … ; Grammar
return root.Descendants("statement") .Select(e => e.FirstElement()) .Where(e => e.Name() != "statementBlock") .Where(e => e.Name() != "labeledStatement") .Where(e => e.Name() != "emptyStatement");
Finder
Finding branches in C
2013/3/20 ICST Testing Tools Track 2013 16
selection_statement : 'if' '(' expression ')' statement | 'switch' '(' expression ')' statement ; iteration_statement : 'while' '(' expression ')' statement | … ;
Grammar
return root.Descendants("selection_statement") .Where(e => e.FirstElement().Value == "if") .Select(e => e.NthElement(2));
Finder
We can consider while as a branch
Finding branches in Python
2013/3/20 ICST Testing Tools Track 2013 17
if_stmt : ’if’ test ’:’ suite (’elif’ test ’:’ suite)* (’else’ ’:’ suite)? ; while_stmt : "while" test ":" suite ("else" ":" suite)? ; Grammar
var ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test")); var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test")); return ifs.Concat(whiles); Finder
Commonality between langs
• Finding statements
– Most of grammars have similar statement nodes
1. Finding statement nodes by a name
2. Excluding unused statement nodes
• Should consider what is a statement
• Finding branches
– Most of grammars have branch statements
1. Finding branch statement nodes by a name
2. Extracting condition nodes (or find directory)
• Should consider what is a branch (if, while, … ?)
2013/3/20 ICST Testing Tools Track 2013 18
Customization
• Changing AST Finders
e.g. Extracting only if statements
2013/3/20 ICST Testing Tools Track 2013 19
var ifs = root.Descendants("if_stmt") .SelectMany(e => e.Elements("test")); var whiles = root.Descendants("while_stmt") .SelectMany(e => e.Elements("test")); return ifs.Concat(whiles); Finder
return root.Descendants("if_stmt") .SelectMany(e => e.Elements("test"));
Finder
Sample applications for FizzBuzz
2013/3/20 ICST Testing Tools Track 2013 20
String answer(int n) { if (n % 15 == 0) { // should return fizzbuzz return "fizzzbuzzz"; } else if (n % 3 == 0) { return "fizz"; } else if (n % 5 == 0) { return "buzz"; } else { return "" + n; } }
Java
@Test void fizz3() { assertEquals("fizz", FizzBuzz.answer(3)); } @Test void fizz6() { assertEquals("fizz", FizzBuzz.answer(6)); } @Test void fizzBuzz15() { assertEquals("fizzbuzz", FizzBuzz.answer(15)); }
JUnit
Apps: Coverage reporter
2013/3/20 ICST Testing Tools Track 2013 21
1: executed 2: executed 3: executed 4: executed 5: 6: 7:
String answer(int n) { if (n % 15 == 0) { // should return fizzbuzz return "fizzzbuzzz"; } else if (n % 3 == 0) { return "fizz"; } else if (n % 5 == 0) { return "buzz"; } else { return "" + n; } }
Java
Statement coverage
57% (4 / 7)
Apps: Fault localization
2013/3/20 ICST Testing Tools Track 2013 22
risk 1: 0.5 2: 1.0 3: 0 4: 0 5: NaN 6: NaN 7: NaN
String answer(int n) { if (n % 15 == 0) { // should return fizzbuzz return "fizzzbuzzz"; } else if (n % 3 == 0) { return "fizz"; } else if (n % 5 == 0) { return "buzz"; } else { return "" + n; } }
Java
Calculate risk metric
We can change definition of
risk metric by writing script
Apps: Test minimization
2013/3/20 ICST Testing Tools Track 2013 23
@Test void fizz3() { assertEquals("fizz", FizzBuzz.answer(3)); } @Test void fizz6() { assertEquals("fizz", FizzBuzz.answer(6)); } @Test void fizzBuzz15() { assertEquals("fizzbuzz", FizzBuzz.answer(15)); }
JUnit
fizz3 and fizz6 are duplicated test cases because they execute
same path
But different test oracles have different detection capabilities ..
Limitation
• Data flow coverage
– OCCF cannot support to implement data flow analyzer due to language independence.
• Supportable languages
– OCCF cannot support languages where we cannot insert instrumentation code.
• Measurement environment
– OCCF works on only .NET Framework and Mono
2013/3/20 ICST Testing Tools Track 2013 24
Conclusion
• Problems – Differences between coverage tools
– High development costs for various langs
• OCCF Solution – Providing common code
– Extracting commonality between langs • Most of ASTs have similar structures
• Please contact us if you want to develop coverage tools with OCCF – We would like to support you
2013/3/20 ICST Testing Tools Track 2013 25