studying the test process in open source and industrial software systems andy zaidman, bart van...
Post on 21-Dec-2015
213 views
TRANSCRIPT
Studying the test process in open source and industrial
software systems
Andy Zaidman, Bart Van Rompaey,
Serge Demeyer, Arie van DeursenTU Delft, University of Antwerp
Andy Zaidman, IPA Herfstdagen 24th November 2008
How does test code* co-evolve?
WHY?
HOW?
(*) Developer tests
Well, why would you…
… want to know the “quality” of the tests?
• When maintaining the system… can you trust the existing tests/test process?
• When judging the overall quality of a software system… is it good enough for our standards?
• When monitoring the test process… Comparing intended with actual process
You might say …
“What about test coverage?”
Nice, but somewhat hollow measure• Easy to keep artificially high• What about boundary testing?• “Exercising versus testing…”• Test process?
How?
Repository mining
• Software developers like to use VCS – e.g., CVS, SVN, Clearcase, MS Visual Sourcesafe, …– Backing up their work– Making it easy to work in groups– Easy to manage releases
• On the other hand… – A wealth of information is stored in there– “It’s like a free museum” of past development activities
and development decisions– Let’s use this!
“Test health”• Determine the long-term quality of your test suite.
• Don’t only look at the “testing product”, but also at the “testing process”– Tells something about the future of the testing (e.g.,
maintenance of the tests)
“Test health” was introduced by Kent Beck
“Test health” VCS views
• Change History View– When are tests adapted?
• Growth History View– Determine impact of changes to production and test
code.
• Test Writing Efficiency View– How much test code for coverage?
3 subject software systems• Checkstyle– Java source-code style checker (and improver)– ~ 6 years development, 2260 commits, 6 developers,
738 classes, 47 kSLOC
• ArgoUML– UML drawing application– ~ 7 years development, 7477 commits, 42 developers,
1533 classes, 130 kSLOC
• 1 industrial system – Software Improvement Group (SIG), Amsterdam,
The Netherlands
Example change history view
Checkstyle Change History
SIG Change History
“CS”
Checkstyle Growth History
%
ArgoUML growth history view
SIG Growth History
Growth History co-evolution scenarios
Checkstyle test writing efficiency
SIG test writing efficiency
Test writing efficiency patterns
Conclusion?
• In our case studies: – Big difference between OS systems and SIG!– SIG• Continuous testing effort• eXtreme Programming + test-driven development
– OS• Sometimes 6 months of no testing• No increase in testing before major release– But sometimes after! Buy 2.1 instead of 2.0!!!
• ArgoUML project was started without testing process in place
Next steps in this research
• Try this on more (diverse) software systems– e.g., OS projects from Apache group
• Try to correlate the testing effort to the number of bugs reported in Bugzilla– More bugs when testing effort is less good, and vice
versa?
• More information?– See our ICST 2008 paper (it’s included in your IPA
Fall Days welcome pack!)– An STVR journal paper is currently in submission
