Comparing model-based and dynamic event-extraction based GUI testing techniques : An empirical study

Gigon Bae, Gregg Rothermel, Doo-Hwan BaeThe Journal of Systems and Software (Volume 97), November 2014, Pages 15 - 46Presented by Xin-Hung Chen, 2015-12-05

2/12

Outline

•Introduction•Preliminaries•GUI testing framework•Test case generation•Empirical study•Conclusion

3/12

Introduction•Graphical user interfaces are pervasive.•One obvious way to test GUIs is with “manual”

testing.▫Time wasting▫High cost

•Researchers have attempted to create GUI testing techniques that automate the processes of generating and executing test cases.

4/12

•Automated GUI testing:▫Model-based GUI testing

More effective at detecting faults Less cost

▫Dynamic event-extraction based GUI testing More effective at covering statements.

Introduction(cont.)

5/12

Introduction(cont.)

•To facilitate this comparison▫ A GUI testing framework

•The overall contributions of this work include the following:▫ A comprehensive experiment comparing the two kinds of GUI testing

techniques.▫ A flexible GUI testing framework that lessens the potential of

implementations.▫ A better understanding of the relative strengths and weaknesses of the

two types of GUI testing techniques.▫ A discussion and better understanding of the various factors affecting

the application of GUI testing techniques.▫ Guidelines for software engineers and researchers on things to

consider.

6/12

Preliminaries

•GUI testing▫A GUI is composed of a set W of widgets that a user

interacts with through a set A of actions. Each action is accompanied by a set V of values.

▫An event e can be represented by a tuple (w, a, v) w W, a A, and v V.∈ ∈ ∈

7/12

Preliminaries(cont.)

•GUI test cases▫A test case t is composed of a sequence of

events (e1, e2, . . ., en). The length of t can be defined as the number of events in t.

▫t = (click button 5, click button +, click button 2, click button =)

8/12

Preliminaries(cont.)

•Event coverage test adequacy▫Event-t-tuple:

Definition: An event-t-tuple (ei, ej, . . ., et) is an ordered tuple of size t of events from E. A set of events E gives rise to |E|t event-t-tuples.

▫t-sequence-cove: Definition: A test suite is t-sequence-cover adequate if it

executes all feasible event-t-tuples by way of an event-t-sequence at least once.

event-2-tuples

9/12

GUI testing framework

10/12

GUI testing framework(cont.)

•Strategy▫Example of google map’s path search

11/12

Test case generation

•Static model-based GUI testing▫Model creation▫Test suite generation▫Test suite execution

•Dynamic event-extraction based GUI testing▫Create test case and executes it at the same

time

12/12

Test case generation(cont.)

•Static model-based GUI testing▫Model creation

Algorithm 1 Input: an AUT, a terminal event list Output: an event flow graph(EFG model) and a set

of initial events I

13/12

Test case generation(cont.)

•Static model-based GUI testing▫Test suite generation

Algorithm 2 Input: an EFG model, a set of initial events I and

a test strength T Output: a test suite TS

14/12

Test case generation(cont.)

•Static model-based GUI testing▫Test suite execution

Algorithm 3 Input: an AUT and a test suite Output: a failure report

15/12

Test case generation(cont.)

•Dynamic event-extraction based GUI testing▫Create and execute test cases

Algorithm 4 Input: an AUT, a test case length K, and a test suite

size S Output: a failure report and a test suite

16/12

Empirical study

•To compare techniques, traditional measures such as fault-detection and elapsed time are necessary but not sufficient.▫How to compare the cost?

17/12

Empirical study(cont.)

•Dependent variables1. Effectiveness in terms of code coverage2. Effectiveness in terms of fault detection3. Cost in terms of numbers of executed events4. Cost in terms of elapsed time5. Event sequence coverage6. GUI State coverage

18/12

Empirical study(cont.)

•Experiment procedure

19/12

Empirical study(cont.)

•Experiment procedure

20/12

Empirical study(cont.)

•Comment

21/12

22/12

23/12

Conclusion

•Model-based GUI testing▫Faults detection▫Less elapsed time

•Dynamic event-extraction GUI testing▫State coverage▫Not effective for short test cases

24/12

Conclusion(cont.)

•The overall contributions of this work include the following:▫ A comprehensive experiment comparing the two kinds of GUI testing

techniques.▫ A flexible GUI testing framework that lessens the potential of

implementations.▫ A better understanding of the relative strengths and weaknesses of the

two types of GUI testing techniques.▫ A discussion and better understanding of the various factors affecting

the application of GUI testing techniques.▫ Guidelines for software engineers and researchers on things to

consider.