qsic2010 shihab
TRANSCRIPT
![Page 1: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/1.jpg)
Prioritizing Unit Test Creation for Test-Driven Maintenance of Legacy
Systems
Emad Shihab, Zhen Ming Jiang, Bram Adams, Ahmed E. Hassan and Robert Bowerman
Queen’s University and Research In MotionCanada
![Page 2: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/2.jpg)
Test Driven Development (TDD)
Write unit test before
writing new code
What about already written code
![Page 3: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/3.jpg)
Test Drive Maintenance (TDM)
Adopting Test Driven Development (TDD)
for Legacy Applications
But time and resources are limited!
![Page 4: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/4.jpg)
Prioritizing Unit Test Creation
Use the rich history of the legacy system to prioritize the writing of unit tests
![Page 5: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/5.jpg)
Avoid the most bugs effectively!
Write unit tests for functions with best Return on Investment (ROI)
How can we avoid the most bugs given limited resources?
![Page 6: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/6.jpg)
Testing Writing Prioritization Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
![Page 7: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/7.jpg)
UsefulnessWas writing the unit test useful?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
Usefulness = 2/3 = 66.67%
![Page 8: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/8.jpg)
POP: Percentage of Optimal Performance How close are we to the optimal performance?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
POP = 8/13 = 61.5%
DE
4 bug fixes
3 bug fixes
![Page 9: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/9.jpg)
Case StudySimulation Parameters
Calibration Period: 6 months
Simulation time: 5 years
List Size: 10 functions
Effort: 100 lines per day
![Page 10: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/10.jpg)
Study SetupExtracting Historical Data
1. Search modification record comments for keywords and bug identifiers
2. Extract source code of modified file(s) and compare to previous version to identify changed functions
3. Combine data from 1 and 2 to identify changed/fixed functions
![Page 11: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/11.jpg)
main() { int a; /*call help*/ helpInfo(); }
helpInfo() { errorString!} main() { int a; /*call help*/ helpInfo(); }
helpInfo(){ int b;} main() { int a; /*call help*/ helpInfo(); }
V1:Undefined func.(Link Error)
V2:Syntax error
V3:Valid code
Mapping Historical Changes to Functions
![Page 12: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/12.jpg)
Study SetupMeasuring the Performance of a Heuristic
Based on a heuristic, generate list of X functions to write unit tests for
Use size of function to measure effort required to write unit test
![Page 13: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/13.jpg)
Test Writing Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
![Page 14: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/14.jpg)
Best Test Writing Heuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
![Page 15: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/15.jpg)
120 240 360 480 600 720 840 960 1080120013201440156016801800192020400
102030405060708090
100
Time (Days)
Usef
ulne
ss (%
) Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)Largest Fixed (LF)
Change Risk (CR)
Random
UsefulnessWas writing the unit test useful?
![Page 16: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/16.jpg)
POP: Percentage of Optimal Performance How close are we to the optimal performance?
120240
360480
600720
840960
10801200
13201440
15601680
18001920
20400
10
20
30
40
50
60
70
80
90
100
Time (days)
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)Random
![Page 17: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/17.jpg)
Largest Fixed (LF)
Largest Modified (LM)
Most Frequently Fixed (MFF)
Most Frequently Modified (MFM)
Most Recently Fixed (MRF)
Change Risk (CR)
Size Risk (SR)
Most Recently Modified (MRM)
Random
0 10 20 30 40 50 60 70 80 90 100
87
84.7
83.8
80
56.9
55
48.8
43.1
27.7
32.4
32.2
22.2
21.8
7
5.5
4.3
4.9
1.7
Overall Performance of HeuristicsPOPUsefulness
![Page 18: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/18.jpg)
5 10 200
5
10
15
20
25
30
35
40
List size
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Effect of Varying ParametersVarying List Size
Most Frequently Modified (MFM)Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)
Random
![Page 19: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/19.jpg)
50 100 2000
5
10
15
20
25
30
35
40
45
50
Effort (lines per day)
Perc
enta
ge o
f opti
mal
per
form
ance
(%)
Effect of Varying ParametersVarying Writing Effort
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Fixed (LF)
Change Risk (CR)
Random
![Page 20: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/20.jpg)
Conclusion
%
![Page 21: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/21.jpg)
![Page 22: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/22.jpg)
TDM ApproachOverview
![Page 23: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/23.jpg)
Performance of Testing Heuristics
![Page 24: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/24.jpg)
Simulation Case Study
Which heuristic is the best?
![Page 25: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/25.jpg)
Simulation Case StudySimulation Parameters
Calibration Period: 6 months
Simulation time: 5 years
List Size: 10 functions
Effort: 100 lines per day
![Page 26: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/26.jpg)
Case Study ResultsModification Heuristics
%
![Page 27: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/27.jpg)
Case Study ResultsFix Heuristics
%
![Page 28: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/28.jpg)
Case Study ResultsSize Heuristics
%
![Page 29: Qsic2010 shihab](https://reader036.vdocuments.pub/reader036/viewer/2022062503/5878f75c1a28ab49608b5b49/html5/thumbnails/29.jpg)
Case Study ResultsRisk Heuristics
%