voxxed days athens - improve your tests with mutation testing

MUTATION TESTINGTO THE RESCUE OF YOUR TESTS

ΝΙΚΟΛΑΣ ΦΡΑΝΚΕΛ

ME, MYSELF AND I

@nicolas_frankel #mutationtesting

By day

• Consultant ($$$)

By night

•Developer

• Blogger

• Book author

• Teacher/trainer

2

HYBRIS, AN SAP COMPANY

@nicolas_frankel #mutationtesting 3

MANY KINDS OF TESTING

@nicolas_frankel #mutationtesting

Unit Testing

Integration Testing

End-to-end Testing

Performance Testing

Penetration Testing

Exploratory Testing

etc.

4

THEIR ONLY SINGLE GOAL

@nicolas_frankel #mutationtesting

Ensure the quality of the production code

5

THE PROBLEM

@nicolas_frankel #mutationtesting

How to check the quality of the testing code?

6

CODE COVERAGE

@nicolas_frankel #mutationtesting

“Code coverage is a measure used to describe the degree to which the source code of a program is tested”

--Wikipediahttp://en.wikipedia.org/wiki/Co

de_coverage

7

MEASURING CODE COVERAGE

@nicolas_frankel #mutationtesting

Check whether a source code line is executed during a test

•Or Branch Coverage

8

COMPUTING CODE COVERAGE

CC =Lexecuted

Ltotal*100

CC: Code Coverage(in percent)

Lexecuted: Number of executed lines of code

Ltotal: Number of total lines of code

@nicolas_frankel #mutationtesting 9

JAVA TOOLS FOR CODE COVERAGE

@nicolas_frankel #mutationtesting

JaCoCo

Clover

Cobertura

etc.

10

100% CODE COVERAGE?

@nicolas_frankel #mutationtesting

“Is 100% code coverage realistic? Of course it is. If you can write a line of code, you can write another that tests it.”

Robert Martin (Uncle Bob)https://twitter.com/unclebobmartin/status/5596662050966732

8

11

ASSERT-LESS TESTING

@Test

public void add_should_add() {

new Math().add(1, 1);

}

@nicolas_frankel #mutationtesting

But, where is the

assert?As long as the Code Coverage is

OK…

12

CODE COVERAGE AS A MEASURE OF QUALITY

@nicolas_frankel #mutationtesting

Any metric can be gamed!

Code coverage is a metric…

⇒ Code coverage can be gamed

• On purpose

• Or by accident

13

CODE COVERAGE AS A MEASURE OF QUALITY

@nicolas_frankel #mutationtesting

Code Coverage lulls you into a false sense of security…

14

THE PROBLEM STILL STANDS

@nicolas_frankel #mutationtesting

Code coverage cannot ensure test quality

• Is there another way?

Mutation Testing to the rescue!

15

THE CAST

@nicolas_frankel #mutationtesting

William Stryker

Original Source Code

Jason Stryker

Modified Source Code

a.k.a “The Mutant”

16

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

@nicolas_frankel #mutationtesting

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

17

STANDARD TESTING

@nicolas_frankel #mutationtesting

✔Execute Test

18

MUTATION TESTING

@nicolas_frankel #mutationtesting

?Execute SAME Test

MUTATION

19

MUTATION TESTING

@nicolas_frankel #mutationtesting

✗

✔Execute SAME Test

Execute SAME Test

Mutant

Killed

Mutant

Survived

20

KILLED OR SURVIVING?

@nicolas_frankel #mutationtesting

Surviving means changing the source code did notchange the test result

• It’s bad!

Killed means changing the source code changed the test result

• It’s good

21

TEST THE CODE

@nicolas_frankel #mutationtesting

✔Execute Test

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

@Test

public void add_should_add() {

new Math().add(1, 1);

}

22

✔Execute Test

SURVIVING MUTANT

@nicolas_frankel #mutationtesting

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

@Test

public void add_should_add() {

new Math().add(1, 1);

}

23

TEST THE CODE

@nicolas_frankel #mutationtesting

@Test

public void add_should_add() {

int sum = new Math().add(1, 1);

Assert.assertEquals(sum, 2);

}

✔Execute Test

public class Math {

public int add(int i1, int i2) {

return i1 + i2;

}

}

24

KILLED MUTANT

@nicolas_frankel #mutationtesting

✗Execute SAME Test

@Test

public void add_should_add() {

int sum = new Math().add(1, 1);

Assert.assertEquals(sum, 2);

}

public class Math {

public int add(int i1, int i2) {

return i1 - i2;

}

}

25

MUTATION TESTING IN JAVA

@nicolas_frankel #mutationtesting

PIT is a tool for Mutation testing

Available as

• Command-line tool

•Ant target

•Maven plugin

26

MUTATORS

@nicolas_frankel #mutationtesting

Mutators are patterns applied to source code to produce mutations

27

PIT MUTATORS SAMPLE

Name Example source Result

Conditionals Boundary > >=

Negate Conditionals == !=

Remove Conditionals foo == bar true

Math + -

Increments foo++ foo--

Invert Negatives -foo foo

Inline Constant static final FOO= 42 static final FOO = 43

Return Values return true return false

Void Method Call System.out.println("foo")

Non Void Method Call long t = System.currentTimeMillis() long t = 0

Constructor Call Date d = new Date() Date d = null;

@nicolas_frankel #mutationtesting 28

IMPORTANT MUTATORS

@nicolas_frankel #mutationtesting

Conditionals Boundary

• Potential serious bug hiding there

•if (foo > bar)

29

IMPORTANT MUTATORS

@nicolas_frankel #mutationtesting

Void Method Call• Assert.checkNotNull()

• connection.close()

30

REMEMBER

@nicolas_frankel #mutationtesting

It’s not because the IDE generates code safely that it will never change

• equals()

• hashCode()

31

ENOUGH TALK…

@nicolas_frankel #mutationtesting 32

SH… HAPPENS

@nicolas_frankel #mutationtesting

False positives Imperfect Sloooooooooooooooow

33

PIT IS IMPERFECT

if (p < 0)

...

// changed condition boundary -> survived:

if (p > 0)

...

return 0;

@nicolas_frankel #mutationtesting 34

PIT IS IMPERFECT

void rebootMachine() {

// removed method call:

checkUserPermissions();

Runtime.getRuntime().exec("reboot");

}

@nicolas_frankel #mutationtesting 35

WHY SO SLOW?

@nicolas_frankel #mutationtesting

Analyze test code

For each class under test

• For each mutator

• Create mutation

• For each mutation

• Run test

• Analyze result

• Aggregate results

36

WORKAROUNDS

Increase number of threads

Set a limited a set of mutators

Limit scope of target classes

Limit number of tests

Limit dependency distance

Use incremental analysis

Don’t bind to the test phase

Use scmMutationCoverage

@nicolas_frankel #mutationtesting 37

INCREMENTAL ANALYSIS

Metadata stored between runs

Mutant will not be checked again, if:

• timed out, and class has not changed• killed, and neither class nor test have changed• survived, and there are no new/changed tests for it

@nicolas_frankel #mutationtesting 38

THE SILVER BULLET?

@nicolas_frankel #mutationtesting

Checks the relevance of your unit tests

Points out potential bugs

39

THE REST IS UP TO YOU

@nicolas_frankel #mutationtesting

Validate the assembled application

• Integration Testing

Check the performance

• Performance Testing

Look out for display bugs

• End-to-end testing

Etc.

40

TESTING IS ABOUT ROI

@nicolas_frankel #mutationtesting

Don’t test to achieve 100% coverage

Test because it saves money in the long run

Prioritize:

• Business-critical code

• Complex code

41

Q&A

@nicolas_frankel #mutationtesting

http://blog.frankel.ch/@nicolas_frankelhttp://frankel.in/https://git.io/vznQK

42