/ 32

Automated Unit Test Generation for Classes with Environment Dependencies

Andrea Arcuri, Gordon Fraser, Juan Pablo Galeotti

ASE 2014

Presented by Yongbae Park

/ 322

Overview• Motivation: low coverage & non-deterministic test results in

automated test generation for a class interacting with environment1. Branches depending on the environment that may not be covered by a

sequence of method calls of a class under test

2. Passing test cases generated by a tool can fail as the environment changes• The environment is inputs from outside of a class that a class cannot control

such as network, file system, system time, and user inputs

• Approach: use a mock library interacting with the virtual environment to increase coverage and produce deterministic test result• Users of a test generation tool do not have to build their own mock library• The mock library provides deterministic inputs from the virtual environment

separated from the real environment

• Evaluation: the mock library increases branch coverage and decreases the number of test cases making non-deterministic results• EvoSuite with the mock library produces test cases whose branch coverage is

increased by 183% and the number of tests with non-deterministic test results is reduced by 99.2%

/ 323

Contents• Background

• How EvoSuite generates test suites using a genetic algorithm

• Motivation• The limitation of EvoSuite in a class with environment dependency

• Approach• Mocking Java standard API to control the environment

• Empirical study• Related Works• Conclusion

/ 324

Background – EvoSuite• Given class under test (CUT), EvoSuite automatically generates a

test suite using a genetic algorithm 1. Create 2 initial test suites by adding method calls randomly and insert the

test suites into current generation

2. Select two test suites from current generation

3. Create 2 new test suites by crossover (exchange test cases of the suites)

4. Modify two test suites (test drivers) from step 3 with mutation operators (remove, insert, change operators)

5. Insert the new two test suites from step 4 to next generation if coverage of the new two test suites are higher than that of parents

6. Repeat 2~5 until next generation has sufficient number of test suites

7. Repeat 2~6 until time limit is reached or all branches are covered

8. Select a test suit with the highest branch coverage and insert assertions by observing differences in execution traces between the original CUT and a mutated CUT

/ 325

Example for EvoSuite• Message class contains 2 members: created meaning

message creation date and text meaning contents of a message• The message creation date is set in the constructor of Message

public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text; } public String getText() { return text;} }

12345678910111213

/ 326

Initial Test Suite Generation• EvoSuite randomly inserts a new statement in empty test cases of two

initial test suites until the length of test cases reaches the maximum length• EvoSuite add a method call whose callee is a method of a class under test or a

method call of an object that is available at the end• A parameter of the created method call is selected from available values, null, or a

random value

• The maximum length for each test case is set by random number

public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text; } public String getText() { return text;} }

123456789

10111213

public class TestSuite1 { public void test0() { //length 3

} public void test1() { //length 2

} }

Message v0 = new Message(“e”);

Message v0 = new Message(null);

Message v1 = new Message(“c”);String v2 = v1.toString();

String v1 = v0.getText();

/ 327

Crossover• EvoSuite selects test suites and from current population, and

moves last test cases of to and first test cases of to where is a random number

public class TestSuite1 { public void test0() { Message v0 = new Message(“e”); Message v1 = new Message(“c”); String v2 = v1.toString();} public void test1() { Message v0 = new Message(null); String v1 = v0.toString();} }

public class TestSuite2 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString(); String v2 = v1.trim();} public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString();} }

public class TestSuite3 { public void test0() { Message v0 = new Message(“e”); Message v1 = new Message(“c”); String v2 = v1.toString();} public void test1() { Message v0 = new Message(“a”); String v1 = v0.toString(); String v2 = v1.trim();} }

public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} public void test1() { Message v0 = new Message(null); String v1 = v0.getText(); String v2 = v0.toString();} }

/ 328

Mutation – Insert Operator• Insert operator adds a new statement at a random position by

using one of the two ways1. Add a method call statement whose callee is a method of a class

under test

2. Add a method call of an object that is available at the random position• A parameter of the created method call is selected from available

values, null, or a random value

public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }

public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); Message v2 = new Message(“b”);} }

public class TestSuite4 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString(); boolean v2 = v1.equals(null);} }

Adding a method call of a class under test

Adding a method call of an available object

/ 329

Mutation – Remove & Change Operator• Remove operator randomly selects a statement in a test case

and remove the statement• Change operator randomly changes a callee or a parameter of a

method invocation statement in a test case• Change operator selects a new callee method that whose return type is

same as the original method• Change operator changes a parameter of a method call into a value which

is available from the previous statements or a random value public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }

public class TestSuite4 { public void test0() { Message v0 = new Message(null); String v1 = v0.getText();} }

public class TestSuite4 { public void test0() { Message v0 = new Message(“a”); String v1 = v0.toString();} }

callee method change parameter change (random value)

/ 3210

Inserting Assertions• EvoSuite adds assertions by observing differences

between an execution trace of the original class under test and that of a mutated version of a class under test• An assertion is added if values of a variable (primitive or String

type) are different in the two execution traces• EvoSuite mutates the original class under test using the insert,

remove, change operators in test generation

/ 3211

Inserting Assertions - Example

public class Message { //original private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text;}…}

123456789

10

public class Message { //mutated private Date created; private String text; public Message(String text) { this.created=new Date();

} public String toString() { return created+”,”+text;}…}

123456789

10

public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString();} }

v1 = “2014.10.02,” v1 = “2014.10.02,null”

Execution result of Message Execution result of mutated Message

assertEquals(v1,”2014.10.02,”)

/ 3212

Generated Test Case• EvoSuite creates a test case with an assertion containing a

string of the created date of the test case• test0() uses a string “2014.10.2,”

• However, the test case will fail when the test case is executed on a different day

public class Message { private Date created; private String text; public Message(String text) { this.created=new Date(); this.text=(text==null?””:text); } public String toString() { return created+”,”+text;} }

123456789

10

public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1,”2014.10.02,”);} }

/ 3213

Motivation• The test result of a class interacting with the environment is non-

deterministic (not reproducible)• The author defines the environment as inputs from outside of a class

• E.g. a state of virtual machine (free & total memory space), a state of operating system (system time, file system, user inputs)

• A test case is unstable if the test case passed when test case is generated but may fail in the next execution with different environment

Class under test

Virtual Machine

Operating System

File system

Time & Date

Memory state User inputs

/ 3214

Environment Mocking• “Mocking” is the replacement real classes with modified

classes that behave deterministically• In the following example, Date is replaced by MockDate that

always returns a fixed value

public class Message { private Date created; private String text; public Message(String text) { this.created=new MockDate(); this.text=(text==null?””:contents); } public String toString() { return created+”,”+text;} }public class MockDate() { public String toString() { return “2014.10.02”;} }

123456789

1011121314

public class TestSuite1 { public void test0() { Message v0 = new Message(null); String v1 = v0.toString(); assertEquals(v1,”2014.10.02,”);} }

/ 3215

Overview of Environment Mocking• A test generation tool with a generic mock library create a

stable test suite with higher branch coverage without user’s (tester) efforts• A mock library of the paper mocks console inputs, file I/O, general

API class of Java standard API• The mock library consist of a custom InputStream class for console

inputs, 11 classes of file I/O (e.g. File), 12 general API classes (e.g. System, Runtime)

• The mock library have helper methods that set the environment in a test case

• The technique replaces standard library with the mock library using Java bytecode instrumentation

• Insert operator of EvoSuite inserts the helper methods in test cases to generate test suites that controls the initial environment

/ 3216

Console Inputs• A customized InputStream called SystemInUtil has

a helper method addInputLine(String) so that a test case can write contents of console inputs• The console contents of SystemInUtil is reset before every test

execution• In instrumentation, System.io in a class under test is changed

into SystemInUtil.io

/ 3217

File I/O• The authors mocked 11 JVM file I/O API classes

• Mock classes are children classes of the mocked classes • The authors overridden the methods of the mocked classes to access

virtual file system instead of real file system of operating system• For example, the authors overridden 37 methods among 52 methods of File class

• Test cases become independent from each other and there is no negative side-effect such as file system corruption

• The authors created helper methods such as appendLineToFile() to control the initial state of the virtual file systemjava.io.File java.io.PrintStream

java.io.FileInputStream java.io.PrintWriter

java.io.FileOutputStream java.util.logging.FileHandler

java.io.RandomAccessFile javax.swing.JFileChooser

java.io.FileReader java.io.FileWriter

javax.swing.filechooser.FileSystemView

/ 3218

General JVM Calls• The authors mocked 12 JVM general API classes to

control the environment such as time and random numberClass name Environment

java.lang.Exception

Stack trace messagejava.lang.Throwable

java.util.logging.LogRecord

java.lang.Thread

java.lang.Runtime Memory usage & the number of processors

java.lang.System

Current system time & datejava.util.Date

java.util.Calendar

java.util.GregorianCalendar

java.lang.Class Reflection (the order of Method objects)

java.lang.MathRandom number

java.util.Random

/ 3219

Stack Trace (1/2)• StackTest has a method getStackTrace() which returns a

stack trace string of Throwable object • The message contains the name of a thread and the name of class that executes

the test case• The test case fails if the name of a thread or test runner is changed

public class StackTest { public static String getStackTrace(Throwable t) { PureStringWriter sw = new PureStringWriter(); PrintWriter pw = new PrintWriter(); t.printStackTrace(pw); return sw.toString();} }public class TestSuite { public void test0() { String result = StackTest.getStackTrace(new Throwable()); assertEqual("Exception in thread \“Thread-0\" java.lang.Throwable\n \tat TestRunner.run(TestRunner.java:120)\n \tat TestSuite.test0(Test.java:10)" , result);} }

123456789101112131415

/ 3220

Stack Trace (2/2)• Stack trace message depends on the string representation

of Thread object and JUnit runner• Stack trace message contains the string representation with thread

id depending on the number of created thread• JUnit runner is a class that executes each test case and bottom of

stack trace message contains the class name of JUnit runner class• A tester can use custom JUnit runner to modify test execution process

• Mocked Thread, Exception, Throwable, and LogRecord create stack trace message deterministically• The mock class of Thread print the string representation

deterministically• The mock class of Thread, Exception, Throwable, and LogRecord does not prints custom JUnit runner class in stack trace

/ 3221

Memory & Processor (1/2)• Performance executes a certain task and measure

execution time and memory usage• getMemoryUsage() returns a string containing the amount of

memory used by the task• The test case fails if free memory space is changed by execution of

a test runner or other test cases

public class Performance { public static String getMemoryUsage() { performTask(); return "Memory: "+(Runtime.totalMemory()-Runtime.freeMemory());} }public class TestSuite { public void test0() { String result = Performance.getMemoryUsage(); assertEqual(“Memory: 341”, result);} }

12345678910

/ 3222

Memory & Processor (2/2)• Runtime provides system specific values such as the

number of processors and memory usage• availableProcessors() returns the number of available

processors• freeMemory(), totalMemory(), and maxMemory() return

memory usage

• availableProcessors(), freeMemory(), totalMemory(), and maxMemory() of a mock class of Runtime return constant values

/ 3223

System Time (1/2)• Message object contains the creation date of a message

• The generated test case contains a string of the day when the test case was created

• The test case fails if the test case is executed on 7next day

public class Message { private Date created; private String msg; public Message(String msg) { created = new Date(); this.msg = msg; } public Date getCreated() { return created;} }public class TestSuite { public void test0() { Message msg = new Message(); assertEqual(“2014.10.02”, msg.getCreated().toString());} }

123456789101112131415

/ 3224

System Time (2/2)• Mock methods and classes of currentTimeMillis(), nanoTime(), Date, Calendar, and GeregorianCalendar returns default time• The mock library has helper methods to set initial system time • Each call of the system time methods increases the time by 1

/ 3225

Reflection (1/2)• getAvailableMethods() returns a string of a list of public

methods• If test0() is executed before test1(), the execution result of test0() is

pass• Otherwise, the execution result of test0() is fail

public class LoadLibrary { public static String getAvailableMethods(Class c) { String result = “”; for (Method each : c.getMethods() ) { result += each.getName() + “\n”; } return result;} }public class TestSuite { public void test0() { String methods = getAvailableMethods(TestSuite.class); assertEqual(“test0\ntest1”, methods); public void test1() { …} }

123456789101112131415

/ 3226

Reflection (2/2)• The order of an array returned by reflection methods is

non-deterministic• The order of an array that is returned by reflection methods

(getClasses(), getMethods()) depends on method invocation order or class loader implementation

• The reflection methods of a mock class of Class return sorted lists of available methods, classes, annotations, etc.

/ 3227

Random Number• random() returns position data in 2D space randomly

• The content of the returned object can be different from the previous execution

• Mock classes of Math and Random return the next random number deterministically

public class Position { public int x, y; public static Position random() { Position result = new Position(); result.x = (int)Math.random(); result.y = (int)Math.random(); return result;} }public class TestSuite { public void test0() { Position p = Position.random(); assertEqual(2, p.x);} }

12345678910111213

/ 3228

Static Field Initialization• A class under test that modifies static field may change

behavior of test cases if the execution order of the test cases is changed• EvoSuite creates unstable test because EvoSuite does not reset

static field nor restart Java virtual machine

• The extended EvoSuite calls static initialization methods of classes whose static fields are modified to reset static field with low overhead• Every writing to a static field is instrumented with a method call that

records the modification• A test suite generated by EvoSuite calls the static initialization

methods before the execution of each test case

/ 3229

Empirical Study• RQ1: does controlling the environment successfully

increase coverage on known cases of environmental interaction?

• RQ2: does controlling the environment resolve known issues of unstable tests?

• RQ3: How do results generalize to the SF100 corpus?

• The authors applied extended EvoSuite to SF100 corpus• SF100 corpus is 100 Java projects randomly selected from

SourceForge• SF100 corpus contains 11,219 Java classes

• EvoSuite created test cases up to 3 minutes for each class

/ 3230

RQ1: Coverage – Setup• The authors manually selected 30 classes from SF100

corpus that interact with the environment• The classes that uses environment-related API are selected

• The authors compared coverage of 6 different configurations• Base: the default EvoSuite• Console: EvoSuite with console input mocking• VFS: EvoSuite with virtual file system• JVM: EvoSuite with general API call mocking• Static: EvoSuite with static field initialization• All: EvoSuite with console input, virtual file system, general API call

mocking and static field initialization

/ 3231

RQ1: Coverage – Experiment Results• The technique increases branch coverage by 183%(=(0.82-0.29)/0.29) in average

• The blue numbers are statistically significant differences at

No. BaseExtended

Console VFS JVM Static All

1 0.10 0.10 0.76 0.10 0.10 0.78

2 0.75 0.75 0.75 0.75 1.00 1.00

3 0.09 0.08 0.53 0.08 0.41 0.80

4 0.26 0.74 0.27 0.25 0.26 0.73

5 0.40 0.40 0.89 0.40 0.40 0.99

6 0.60 0.60 0.60 0.61 0.99 0.99

7 0.20 0.20 0.77 0.20 0.20 0.77

8 0.29 0.29 0.40 0.28 0.29 0.40

9 0.94 0.94 0.94 0.95 1.00 1.00

10 0.83 0.83 0.83 0.95 0.83 0.94

11 0.12 0.12 0.71 0.12 0.12 0.70

12 0.08 0.68 0.09 0.09 0.11 0.64

13 0.24 0.24 0.24 0.24 0.79 0.86

14 0.12 0.12 0.12 0.12 0.12 0.93

15 0.33 0.33 0.80 0.33 0.33 0.80

No. BaseExtended

Console VFS JVM Static All

16 0.22 0.22 0.74 0.22 0.22 0.73

17 0.00 0.00 0.87 0.00 0.00 0.87

18 0.00 0.00 1.00 0.00 0.00 1.00

19 0.30 0.33 0.77 0.30 0.69 0.79

20 0.00 0.00 1.00 0.00 0.00 1.00

21 0.79 0.79 0.77 0.67 0.88 0.85

22 0.67 0.68 0.69 0.61 0.99 0.96

23 0.08 0.08 0.99 0.08 0.08 0.99

24 0.25 0.25 0.87 0.25 0.25 0.99

25 0.07 0.07 0.56 0.07 0.07 0.56

26 0.22 0.66 0.22 0.22 0.22 0.63

27 0.28 0.62 0.30 0.27 0.25 0.64

28 0.03 0.03 0.56 0.03 0.03 0.54

29 0.34 0.34 0.79 0.34 0.34 0.80

30 0.20 0.20 0.79 0.20 0.20 0.79

Avg. 0.29 0.35 0.65 0.29 0.37 0.82

/ 3232

RQ2: Unstable Tests – Setup• The authors manually selected 30 classes from SF100

corpus from which the default EvoSuite creates unstable tests • There are 2 classes that are used in both RQ1 and RQ2

• The authors compared coverage of 6 different configurations same as the experiments in RQ1

/ 3233

RQ2: Unstable Tests – Experiment Results• The technique reduces the number of unstable tests per class by 99.2%(=(2.43-

0.02)/2.43)• The blue numbers are statistically significant differences at

No. BaseExtended

Console VFS JVM Static All

1 2.72 2.52 2.52 2.49 0.00 0.00

2 1.14 1.16 1.20 2.06 0.00 0.00

3 2.99 3.04 3.17 3.21 0.14 0.13

4 1.27 1.28 1.35 1.74 0.00 0.00

5 2.37 2.36 2.33 2.19 0.00 0.07

6 4.20 4.19 4.24 4.15 1.55 0.01

7 1.33 1.40 1.39 0.00 1.40 0.00

8 0.64 0.75 0.79 0.82 0.06 0.08

9 0.84 0.98 0.75 0.00 0.77 0.00

10 7.16 7.09 6.93 0.03 6.94 0.00

11 0.61 0.62 0.61 0.00 0.62 0.00

12 2.50 2.52 2.52 2.38 0.00 0.00

13 5.53 5.57 5.47 5.87 0.00 0.00

14 3.73 3.66 3.41 0.02 1.73 0.00

15 6.86 6.79 7.50 4.41 7.05 0.30

No. BaseExtended

Console VFS JVM Static All

16 1.75 1.73 1.47 1.57 0.00 0.00

17 1.19 1.23 1.03 0.00 1.17 0.00

18 2.59 2.87 2.55 0.09 2.27 0.11

19 2.49 2.39 2.93 0.02 1.62 0.00

20 1.91 1.90 1.87 0.00 1.77 0.00

21 3.96 3.87 2.40 3.81 0.00 0.00

22 0.93 0.93 0.78 0.00 0.80 0.03

23 1.82 1.59 1.28 0.00 1.55 0.00

24 3.26 3.67 3.66 0.00 3.38 0.00

25 1.83 1.76 1.79 0.12 1.60 0.16

26 2.13 2.12 2.21 0.00 2.19 0.00

27 1.91 1.86 2.18 1.79 0.01 0.00

28 1.63 2.01 1.87 0.86 1.22 0.00

29 1.22 1.22 1.22 0.00 1.19 0.00

30 0.57 0.54 0.57 0.67 0.00 0.00

Avg. 2.43 2.45 2.40 1.27 1.30 0.02

/ 3234

RQ3: Generalization• The authors compared the default EvoSuite (base) and the

extended EvoSuite (all) by applying both of them to all 11,219 classes of SF100 corpus

• The branch coverage increased by 1.8% (=(77.9%-76.5)/76.5)• The default EvoSuite achieved 76.5% of branch coverage and the

extended EvoSuite achieved 77.9%• The extended EvoSuite achieved 2,803 additional branches

• The number of classes from which EvoSuite creates unstable tests is reduced by 53.8% (=(1,452-671)/1,452)• The default EvoSuite created unstable tests from 1,452 classes while

the extended EvoSuite created unstable tests from 671 classes

/ 3235

Related Works• Mocking frameworks help developers write mock classes

of user classes, not class of the standard library• The mocking frameworks such as Moles (C#), Jmock (Java)

provide mechanisms to create mock class and replace the original classes with the mock classes

• The developers should change mock classes as the original class changes

• The paper aims mocking Java standard API which does not change frequently• Once mock library is built, the mock library can be applied to any

class under test

/ 3236

Conclusion• Mocking library interacting with the virtual environment

increases coverage and reduces unstable tests in automated unit test generation• The modified EvoSuite creates test cases that controls initial

environment to increase branch coverage• The generated test cases are stable because interactions with the

virtual environment is deterministic

• The authors shown the empirical results that using mocking library increase branch coverage and reduces unstable tests from EvoSuite for 100 Java projects