qsic2010 shihab
Post on 13-Jan-2017
75 Views
Preview:
TRANSCRIPT
Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy
Systems
Emad Shihab, Zhen Ming Jiang, Bram Adams, Ahmed E. Hassan and Robert Bowerman
Queen’s University and Research In MotionCanada
Test Driven Development (TDD)
Write unit test before
writing new code
What about already written code
Test Drive Maintenance (TDM)
Adopting Test Driven Development (TDD)
for Legacy Applications
But time and resources are limited!
Prioritizing Unit Test Creation
Use the rich history of the legacy system to prioritize the writing of unit tests
Avoid the most bugs effectively!
Write unit tests for functions with best Return on Investment (ROI)
How can we avoid the most bugs given limited resources?
Testing Writing Prioritization Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
UsefulnessWas writing the unit test useful?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
Usefulness = 2/3 = 66.67%
POP: Percentage of Optimal Performance How close are we to the optimal performance?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
POP = 8/13 = 61.5%
DE
4 bug fixes
3 bug fixes
Case StudySimulation Parameters
Calibration Period: 6 months
Simulation time: 5 years
List Size: 10 functions
Effort: 100 lines per day
Study SetupExtracting Historical Data
1. Search modification record comments for keywords and bug identifiers
2. Extract source code of modified file(s) and compare to previous version to identify changed functions
3. Combine data from 1 and 2 to identify changed/fixed functions
main() { int a; /*call help*/ helpInfo(); }
helpInfo() { errorString!} main() { int a; /*call help*/ helpInfo(); }
helpInfo(){ int b;} main() { int a; /*call help*/ helpInfo(); }
V1:Undefined func.(Link Error)
V2:Syntax error
V3:Valid code
Mapping Historical Changes to Functions
Study SetupMeasuring the Performance of a Heuristic
Based on a heuristic, generate list of X functions to write unit tests for
Use size of function to measure effort required to write unit test
Test Writing Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
Best Test Writing Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
120 240 360 480 600 720 840 960 1080120013201440156016801800192020400
102030405060708090
100
Time (Days)
Usef
ulne
ss (%
) Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)Largest Fixed (LF)
Change Risk (CR)
Random
UsefulnessWas writing the unit test useful?
POP: Percentage of Optimal Performance How close are we to the optimal performance?
120240
360480
600720
840960
10801200
13201440
15601680
18001920
20400
10
20
30
40
50
60
70
80
90
100
Time (days)
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)Random
Largest Fixed (LF)
Largest Modified (LM)
Most Frequently Fixed (MFF)
Most Frequently Modified (MFM)
Most Recently Fixed (MRF)
Change Risk (CR)
Size Risk (SR)
Most Recently Modified (MRM)
Random
0 10 20 30 40 50 60 70 80 90 100
87
84.7
83.8
80
56.9
55
48.8
43.1
27.7
32.4
32.2
22.2
21.8
7
5.5
4.3
4.9
1.7
Overall Performance of HeuristicsPOPUsefulness
5 10 200
5
10
15
20
25
30
35
40
List size
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Effect of Varying ParametersVarying List Size
Most Frequently Modified (MFM)Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)
Random
50 100 2000
5
10
15
20
25
30
35
40
45
50
Effort (lines per day)
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Effect of Varying ParametersVarying Writing Effort
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)
Random
Conclusion
%
TDM ApproachOverview
Performance of Testing Heuristics
Simulation Case Study
Which heuristic is the best?
Simulation Case StudySimulation Parameters
Calibration Period: 6 months
Simulation time: 5 years
List Size: 10 functions
Effort: 100 lines per day
Case Study ResultsModification Heuristics
%
Case Study ResultsFix Heuristics
%
Case Study ResultsSize Heuristics
%
Case Study ResultsRisk Heuristics
%
top related